Dynamic vehicle warning signal emission

ABSTRACT

A vehicle computing system may implement techniques to dynamically modify warning signals from a vehicle to ensure that an object (e.g., dynamic object) is notified of the vehicle operation. A vehicle computing system may emit a first warning signal including an audio and/or visual signal and may detect an object reaction to the first warning signal. Based on a determination that the object reaction does not substantially alter the ability for the vehicle to overcome the object, the vehicle computing system may modify a frequency, volume, luminosity, color, shape, motion, etc. of the first warning signal to emit a second warning signal. The vehicle computing system may continually modify warning signals until the object reacts according to an expected reaction or becomes irrelevant to the vehicle.

BACKGROUND

Vehicles in operation today are often equipped with horns that enable anoperator of a vehicle to call attention to the vehicle, such as to warnothers of a potential hazard in an environment. Conventional vehiclehorns are configured to emit a sound at a particular frequency andvolume. However, the particular frequency and/or volume of the vehiclehorn may often be insufficient to get the attention of a pedestrian,such as one listening to music via headphones or one who is hard ofhearing. As such, the vehicle horn may be ineffective in warning othersof a potential hazard.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 is an illustration of an environment, in which a dynamic warningsignal system may be used by an autonomous vehicle to warn an object ofa potential conflict between the vehicle and an object in theenvironment, in accordance with examples of the disclosure.

FIG. 2 is an illustration of a process for modifying warning signalsemitted by a vehicle based at least in part on a detected objectreaction to the emitted warning signals.

FIG. 3 is an illustration of an environment in which a vehicle emitssignals based on a determination that an object is blocking a vehiclepath of the vehicle, the signals including a warning signal to alert theobject of the blocking and an object path signal to indicate a potentialobject path for the object.

FIG. 4 is a block diagram of an example system for implementing thetechniques described herein.

FIG. 5 depicts an example process for emitting different signals to warnan object of a potential conflict between a vehicle and the object.

FIG. 6 depicts another example process for emitting warning signalsbased at least in part on a location of a vehicle and detection of anobject that is relevant to the vehicle.

FIG. 7 depicts an example process for emitting at least one of a warningsignal or a routing signal based on a determination that an object isblocking a path of a vehicle.

DETAILED DESCRIPTION

This disclosure is directed to techniques for improving vehicle warningsystems. The vehicle warning systems may be configured to emit a soundand/or a light to warn objects (e.g., dynamic object) in an environmentproximate the vehicle of a potential conflict with the vehicle. Thevehicle may include an autonomous or semi-autonomous vehicle. Theobjects may include pedestrians, bicyclists, animals (e.g., dogs, cats,birds, etc.), other vehicles (e.g., cars, trucks, motorcycles, mopeds,etc.), or any other object that may potentially cause a conflict (e.g.,collision) with the vehicle. A vehicle computing system may beconfigured to identify an object in the environment and determine that apotential conflict between the vehicle and the object may occur. Thevehicle computing system may emit a first signal to warn the object ofthe potential conflict and, based on a determination that an objectreaction did not substantially match an expected reaction, emit a second(different) signal. The vehicle computing system may continue to modifywarning signals until the object reacts according to the expectedreaction or the object is no longer relevant to the vehicle (e.g.,potential of collision no longer exists), thereby maximizing safeoperation of the vehicle.

The vehicle computing system may be configured to identify objects inthe environment. In some examples, the objects may be identified basedon sensor data from sensors (e.g., cameras, motion detectors, lidar,radar, etc.) of the vehicle. In some examples, the objects may beidentified based on sensor data received from remote sensors, such as,for example, sensors associated with another vehicle or sensors mountedin an environment that are configured to share data with a plurality ofvehicles. In various examples, the vehicle computing system may beconfigured to determine classifications associated with the objects,such as whether the objects are pedestrians, bicyclists, animals, othervehicles, or the like.

The vehicle computing system may be configured to emit a first warningsignal to alert one or more objects in the environment of the vehiclepresence and/or operation. The first warning signal may include an audiosignal and/or a light signal. The first warning signal may include afirst set of characteristics, such as frequency, volume, luminosity,color, shape, motion, or the like. In various examples, the firstwarning signal may be emitted based on a detection of an object in theenvironment and/or features associated with the detection. In suchexamples, the features associated with the detection may include adistance between the vehicle and the object, a relative speed betweenthe vehicle and the object, and the like. For example, the vehiclecomputing system may detect a bicyclist on the road and may determinethat the bicyclist may not hear the vehicle approaching from behind. Thevehicle computing system may emit a warning signal toward the bicyclist,such as to warn the bicyclist of the vehicle's approach so that thebicyclist does not swerve or otherwise maneuver into the road.

In some examples, the first warning signal may be emitted based on aclassification, subclassification (e.g., age, height, etc.), and/oradditional features associated with the detected object. In suchexamples, the vehicle computing system may determine the classification,subclassification, and/or additional features associated with thedetected object and may determine the first set of characteristicsassociated with the first warning signal based on the classification,subclassification, and/or additional features. For example, a firstwarning signal generated to warn a pedestrian of the vehicle operationmay include a lower volume than a first warning signal generated to warnthe bicyclist described in the example above. For another example, afirst warning signal generated to get the attention of an operator of acar may include a higher volume than a first warning signal generated toget the attention of an operator of a motorcycle. For yet anotherexample, a first warning signal generated for a pedestrian wearingheadphones may include a first frequency and a first warning signalgenerated for a pedestrian that is looking at (e.g., a directionassociated with) the vehicle may include a second frequency.

In various examples, the first warning signal may be emitted based on alocation associated with the vehicle, such as a location associated withpedestrians, bicyclists, or other objects (e.g., school zone, proximityto a playground, construction zone, etc.). In some examples, the firstwarning signal may be emitted based on a speed associated with thevehicle (e.g., less than 15 miles per hour, less than 30 kilometers perhour, etc.). In some examples, the first warning signal may include anelectric vehicle warning sound, such as that required by law and/orregulation.

In some examples, the vehicle computing system may cause the firstwarning signal to be emitted based on a determination that a detectedobject is relevant to the vehicle (e.g., a potential conflict betweenthe vehicle and object may exist, object may potentially slow forwardprogress of the vehicle). In various examples, the vehicle computingsystem may be configured to determine relevance of an object utilizingthe techniques described in U.S. patent application Ser. No. 16/193,945,filed Nov. 16, 2018 and entitled “Dynamic Sound Emission for Vehicles,”the entire contents of which are incorporated herein by reference. Insome examples, the determination of object relevance may be based on alocation associated with the object being within a threshold distance ofa path of the vehicle. In such examples, the path may correspond to adrivable surface over which the vehicle plans to travel from a firstlocation to a destination. In some examples, the determination of objectrelevance may be based on a potential trajectory of the objectintersecting a trajectory associated with the vehicle (e.g., trajectoryassociated with the vehicle path). In such examples, the vehiclecomputing system may determine the potential object trajectory based onthe sensor data.

In various examples, the trajectory and/or intent of an object may bedetermined utilizing techniques described in U.S. Pat. No. 10,414,395,issued Sep. 17, 2019 and entitled “Feature-Based Prediction,” the entirecontents of which are incorporated herein by reference. For example, thevehicle computing system may detect a pedestrian jaywalking in the roadahead of the vehicle. The vehicle computing system may determine thatthe pedestrian trajectory may conflict with the vehicle trajectory, suchthat, absent a modification to one or both trajectories, a collisionbetween the vehicle and the pedestrian could occur. The vehiclecomputing system may cause the first warning signal to be emitted towarn the pedestrian of the vehicle operation on the road. In someexamples, the vehicle computing system may cause the first warningsignal to be emitted concurrently with or immediately prior to modifyingthe vehicle trajectory (e.g., yielding to the pedestrian), such as tomaximize safe operation of the vehicle.

In various examples, the vehicle computing system may determine anobject reaction to the first warning signal, based on sensor data. Insome examples, the reaction may include a change in the objecttrajectory (e.g., speed increase, speed decrease, direction of travelaway from the vehicle, etc.), a movement of the head and/or shoulders ofthe object, a gesture (e.g., a wave, etc.), a foot placement of theobject, a positional adjustment to an item the object holds (e.g.,adjusting a position of an electronic device, book, magazine, or otheritem), and/or any other movement indicative of an object reacting thefirst warning signal.

In various examples, the vehicle computing system may compare the objectreaction to an expected reaction (also referred to generally as anobject action) associated with the first warning signal (also referredto generally as a first signal). In various examples, the vehiclecomputing system may be configured to determine the expected reactionbased on one or more characteristics of the first warning signal (e.g.,volume, frequency, luminosity, color, motion (e.g., animated motion,light sequencing, etc.), shape of the signal, etc.) and/or dataassociated with the object (e.g., object attribute (e.g.,classification, position (e.g., facing/moving toward the vehicle,facing/moving away from the vehicle, etc.), distance from the vehicle,trajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.). In some examples, the vehicle computing system mayaccess a database of expected reactions to determine the expectedreaction associated with the first warning signal. In such examples, theexpected reactions in the database may be stored based at least in parton the data associated with the object and/or characteristic(s) of thefirst warning signal. In various examples, the vehicle computing systemmay determine an expected reaction utilizing machine learningtechniques. In such examples, a model may be trained utilizing trainingdata including a plurality of warning signals and detected reactionsthereto.

Based on the comparison between the object reaction and the expectedreaction, the vehicle computing system may determine whether the objectreacted as expected (e.g., whether a substantial match exists betweenthe object reaction and the expected reaction). Responsive to adetermination that the object reaction substantially matches theexpected reaction, the vehicle computing system may store the encounter(e.g., data associated with first warning signal and the objectreaction) in the database. In some examples, the database may be usedfor future object reaction comparisons, such as to increase a confidencein a reaction to the first warning signal, to train the machine learnedmodel, or the like.

In various examples, the determination of a substantial match betweenthe object reaction and the expected reaction may include a match of athreshold number of actions (e.g., one matching actions, two matchingactions, etc.), a threshold percentage of actions (e.g., 90%, 50%,etc.), or the like. In some examples, the substantial match may bedetermined based on a threshold match and/or threshold differencebetween the object reaction and the expected reaction. The actions mayinclude trajectory modifications (e.g., increase in speed, decrease inspeed, change in direction of travel, etc.), body movements (e.g., footplacement, head rotation, shoulder movement, etc.), gestures, or thelike. For example, an expected reaction to the first warning signal mayinclude a head and/or shoulder movement and a positional adjustment toan electronic device the object holds. The object reaction may include ahead movement toward the vehicle. Based on a match of at least the headmovement, the vehicle computing system may determine that the objectreaction and the expected reaction substantially match. For anotherexample, the vehicle computing system may determine that an objectreaction matches an expected reaction at 75%, with a threshold match at65%. Based on a determination that the percentage of the match meets orexceeds the threshold match, the vehicle computing system may determinethat the object reaction substantially matches the expected reaction.

In some examples, the determination of a substantial match between theobject reaction and the expected reaction may include determining that amodification to an object trajectory meets or exceeds a thresholdmodification. In some examples, the threshold modification may include amodification that renders the object irrelevant to the vehicle (e.g.,does not impede progress of the vehicle, no potential for conflict,etc.). In such examples, based at least in part on determining themodification, the vehicle computing system may cause the vehicle toproceed along a vehicle trajectory (e.g., at a planned speed, direction,etc.). In some examples, the threshold modification may include a changein speed and/or direction associated with the object trajectory (e.g.,45 degrees, 90 degrees, etc.).

Responsive to a determination that the object reaction did notsubstantially match (e.g., less than the threshold number of actions,percentage match, etc.), the vehicle computing system may determine thatthe object did not react according to an expected reaction. In suchexamples, the vehicle computing system may determine that the objectremains unaware of the vehicle operation and/or presence of the vehiclein the environment. Based on a determination that the object did notreact according to the expected reaction, the vehicle computing systemmay emit a second warning signal. In some examples, the second warningsignal may include a signal of a different modalit(ies) (e.g., light,sound, etc.) than the first warning signal. For example, the firstwarning signal may include a sound emission and the second warningsignal may include a light emission.

In some examples, the second warning signal may include a signal of asame modalit(ies) as the first warning signal. In such examples, thevehicle computing system may modify a frequency, volume, luminosity,color, shape, motion, and/or other characteristic of the first warningsignal to generate the second warning signal. For example, based on adetermination that a detected object did not react according to anexpected reaction to a first warning signal including a first frequencyemitted at 50 decibels, the vehicle computing system may cause a secondwarning signal including a second frequency to be emitted at 70decibels. For another example, based on a determination that a detectedobject did not react to a first warning signal including a red and greenlight emission, the vehicle computing system may cause a second warningsignal including a yellow and blue light to be emitted. However, it isunderstood that the specific volumes and colors in the aforementionedexamples are merely for illustrative purposes, and other signalcharacteristics (e.g., volume, frequency, luminosity, color, shape,motion, etc.) are contemplated herein.

In various examples, the vehicle computing system may compare a secondobject reaction to a second expected reaction associated with the secondwarning signal. Responsive to a determination that the second objectreaction substantially matches the second expected reaction, the vehiclecomputing system may store data associated with the object reactionand/or the second warning signal in the database of object reactions. Asdiscussed above, in some examples, the database may be used for futureobject reaction comparisons, such as to increase a confidence in areaction to a warning signal, to train the machine learned model, or thelike.

Responsive to a determination that the second object reaction does notsubstantially match the second expected reaction, the vehicle computingsystem may cause a third warning signal to be emitted, the third warningsignal being different from the first warning signal and the secondwarning signal (e.g., different modality, different characteristics,etc.). Continuing the example from above, based on a determination thata second object reaction to the second frequency emitted at 70 decibelsdoes not substantially match a second expected reaction, the vehiclecomputing system may cause a third frequency to be emitted at 90decibels.

In various examples, the vehicle computing system may continue to modify(e.g., iteratively modify) emitted warning signals until an objectreaction substantially matches an expected reaction to the warningsignal. In various examples, the vehicle computing system may continueto modify emitted warning signals based on a determination that theobject is relevant to the vehicle. In such examples, a modified warningsignal may be emitted based on a determination that the detected objectis relevant to the vehicle. In some examples, the vehicle computingsystem may be configured to continually and/or periodically (e.g., every0.1 seconds, 1.0 seconds, prior to generating a modified warning signal,etc.) determine whether the detected object is relevant to the vehicle.For example, a vehicle computing system may determine that a detectedobject did not react to the second warning signal according to thesecond expected reaction. However, prior to emitting a third warningsignal, the vehicle computing system may determine that the detectedobject is behind the vehicle and traveling in a different direction fromthe vehicle. As such, the vehicle computing system may determine thatthe detected object is no longer relevant to the vehicle and maydetermine to not emit the third warning signal.

In various examples, based on a determination that the detected objectreacted in accordance with an expected reaction and/or that the detectedobject is irrelevant to the vehicle, the vehicle computing system maycease emitting a warning signal. In some examples, based on thedetermination that the detected object reacted in accordance with theexpected reaction and/or the detected object is irrelevant to thevehicle, the vehicle computing system may cause the first warning signalto be emitted. In such examples, the first warning signal may include abaseline warning signal emitted to alert nearby objects of the vehiclepresence and/or operation. For example, and as discussed above, thebaseline warning signal may include an electric vehicle warning sound,such as that required by law and/or regulation. For another example, thebaseline warning signal may include a sound and/or light emitted basedon a location associated with the vehicle.

In addition to providing a warning signal to alert an object of thepresence and/or operation of the vehicle, the vehicle computing systemmay be configured to generate a routing signal for the object in theenvironment. In various examples, the routing signal may include aproposed route for the object to take to avoid a conflict (e.g.,collision, blockage, etc.) with the vehicle. In various examples, thevehicle computing system may generate the routing signal based on adetermination that the object is a blocking object. In such examples,the vehicle computing system may determine that the object is blocking apath of the vehicle. The object may be blocking the path of the vehiclebased on a determination that the object is stopped at a location thatat least partially blocks forward progress of the vehicle toward adestination. In some examples, the vehicle computing system maydetermine that the object is the vehicle based on a determination thevehicle may be unable to proceed toward the destination while stayingwithin the confines of a drivable corridor (e.g., drivable surface overwhich the vehicle plans to travel along the path). For example, theobject may be stopped in an intersection in the path of the vehicle,such that the vehicle is unable to proceed through the intersection.

In various examples, based on a determination that the object is ablocking object, the vehicle computing system may be configured toidentify potential routing options for the blocking object. A potentialrouting option may include a clear (e.g., unoccupied) path the blockingobject may follow to move out of the path of the vehicle. In someexamples, the potential routing option may include an area into whichthe blocking object may move. In some examples, the area may includethat which an operator of the blocking object is unable to view, such asdue to another object being located between the blocking object and thearea. Using the example from above, the blocking object may be turningleft in an intersection in front of the vehicle and may be stopped in afirst lane in an intersection behind a delivery vehicle. The blockingobject may be unable to see that an area beyond the intersection in asecond lane is unoccupied and therefore the blocking object may beunaware of the area into which the blocking object may move to clear theintersection. The vehicle computing system may be configured to identifythe area into which the blocking object may move.

In various examples, based on the identification of the area into whichthe blocking object may move (e.g., clear path the blocking object mayfollow to move out of the path of the vehicle), the vehicle computingsystem may cause a routing signal to be emitted. The routing signal mayindicate to the operator of the blocking object that the area is clear.In some examples, the routing signal may include a light emitted in thedirection of the area, an arrow, or other means by which the vehiclecomputing system may communicate the clear area into which the blockingobject may move.

The techniques described herein may substantially improve the safeoperation of autonomous and semi-autonomous vehicle operating in anenvironment. An increasing number of pedestrians, bicyclists, scooterriders, and the like operate on drivable surfaces, often while listeningto music, podcasts, or the like via headphones. The sounds emitted viathe headphones may drown out the sound of the vehicles operating nearby,thereby rendering the people unaware of the presence and/or operation ofthe vehicles, even those vehicles emitting an electric vehicle warningsound. To increase awareness and thus safety of the autonomous and/orsemi-autonomous vehicles, the techniques described herein recognize thatan object is not reacting to a first warning signal emitted by a vehicleand adjust one or more characteristics of the first warning signal in anattempt to alert the object of the vehicle operation and/or presence.The vehicle computing system may continue to modify (e.g., iterativelymodify) the warning signals until the object reacts according to anexpected reaction or is no longer relevant to the vehicle, therebymaximizing the safe operation of the vehicle in the environment.

The techniques described herein may be implemented in a number of ways.Example implementations are provided below with reference to thefollowing figures. Although discussed in the context of an autonomousvehicle, the methods, apparatuses, and systems described herein may beapplied to a variety of systems (e.g., a sensor system or a roboticplatform), and are not limited to autonomous vehicles. In anotherexample, the techniques may be utilized in an aviation or nauticalcontext, or in any system using machine vision (e.g., in a system usingimage data). Additionally, the techniques described herein may be usedwith real data (e.g., captured using sensor(s)), simulated data (e.g.,generated by a simulator), or any combination of the two.

FIG. 1 is an illustration of an environment 100, in which one or morecomputing systems 102 of an autonomous vehicle 104 (e.g., vehicle 104)may utilize a dynamic warning signal system to alert one or more objects106 of a presence and/or operation of the vehicle 104 in the environment100. The computing system(s) 102 may detect the object(s) 106 based onsensor data captured by one or more sensors 108 of the vehicle 104and/or one or one or more remote sensors (e.g., sensors mounted onanother vehicle 104 and/or mounted in the environment 100, such as fortraffic monitoring, collision avoidance, or the like). The sensor(s) 108may include data captured by lidar sensors, radar sensors, ultrasonictransducers, sonar sensors, location sensors (e.g., GPS, compass, etc.),inertial sensors (e.g., inertial measurement units (IMUs),accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB,IR, intensity, depth, time of flight, etc.), microphones, time-of-flightsensors, environment sensors (e.g., temperature sensors, humiditysensors, light sensors, pressure sensors, etc.), and the like.

In some examples, the sensor data can be provided to a perceptioncomponent 110 configured to determine a classification 112 associatedwith the object(s) 106 (e.g., car, truck, pedestrian, bicycle,motorcycle, animal, etc.). In various examples, the perception component110 may determine an object classification 112 based on one or morefeatures associated with the object(s) 106. The features may include asize (e.g., width, height, depth, etc.), shape (e.g., geometry,symmetry, etc.), and/or other distinguishing features of the object(s)106. For example, the perception component 110 may recognize a sizeand/or shape of an object 106, such as object 106(1), corresponds to apedestrian and a size and/or shape of another object 106, such as object106(2), corresponds to a cyclist.

In various examples, based in part on a detection of one or more objects106 in the environment 100, a warning signal component 114 of thecomputing system(s) 102 may generate and/or cause a first warning signalto be emitted in order to alert the object(s) 106 of the vehicle 104presence and/or operation. The first warning signal may include an audiosignal and/or a visual signal. The first warning signal may include afirst set of characteristics, such as frequency, volume, luminosity,color, shape, motion, or the like. In some examples, the first set ofcharacteristics may include a pre-determined set of characteristics. Insuch examples, the first warning signal may include a baseline warningsignal associated with alerting objects 106 of the presence and/oroperation of the vehicle 104. For example, the first warning signal mayinclude an electric vehicle warning sound including a pre-determinedfrequency and emitted at a pre-determined volume.

In various examples, the first set of characteristics may be determineddynamically, such as based on one or more real-time conditionsassociated with the environment 100. The real-time conditions mayinclude data associated with the object 106 (e.g., object attribute(e.g., classification, position (e.g., facing/moving toward the vehicle,facing/moving away from the vehicle, etc.), distance from the vehicle,trajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.), environmental factors (e.g., noise level in theenvironment 100, amount of traffic, road conditions, etc.), weatherconditions (e.g., rain, snow, hail, wind, etc.), vehicularconsiderations (e.g., speed, passengers in the vehicle 104, etc.), andthe like. For example, the first set of characteristics associated witha first warning signal generated for a pedestrian wearing headphones mayinclude a first frequency and a first set of characteristics associatedwith a first warning signal generated for a pedestrian that is lookingin a direction associated with the vehicle may include a secondfrequency.

In various examples, the frequency (e.g., one or more frequencies) ofthe first warning signal may include a frequency (or set/range offrequencies) that is perceptible to the object 106, such as based on theclassification 112 of the object 106. For example, the warning signalcomponent 114 may determine that an object 106 is a dog. Based in parton the classification 112 as a dog, the warning signal component 114 maydetermine to emit a first warning signal at a frequency detectable todogs and not humans, such as to cause the dog to avoid the vehicle 104and/or the vehicle path. In various examples, one or morecharacteristics of the first set of characteristics (e.g., volume and/orvolume range, one or more frequencies, luminosities, shapes, motions,and/or color(s)) may be determined based on an urgency of the warning(e.g., low urgency (e.g., alert), medium urgency ((e.g., caution), highurgency (e.g., warning)), a likelihood of conflict between the vehicle104 and the object 106, a message to be conveyed to the object 106(e.g., the vehicle 104 is approaching, please stop, trajectories arerapidly converging, etc.).

In various examples, one or more characteristics of the first set ofcharacteristics may be determined based on an object activity (e.g., adetected distraction associated with the object 106). The detecteddistraction may include the tactile use of a mobile phone, adetermination that the potentially conflicting object is engaged in aconversation (e.g., with another object proximate to the object 106, ona mobile phone, or the like), a determination that the object 106 iswearing headphones, earmuffs, ear plugs, or any other device configuredto fit in or around an auditory canal.

In some examples, the one or more characteristics of the first set ofcharacteristics may be determined based on weather conditions in theenvironment. The weather conditions may include rain, wind, sleet, hail,snow, temperature, humidity, large pressure changes, or any otherweather phenomenon which may affect an auditory perception of an object106 in the environment 100. In various examples, the one or morecharacteristics of the warning signal may be determined based on roadconditions in the environment. The road conditions may include asmoothness of road surface (e.g., concrete, asphalt, gravel, etc.), anumber of potholes, uneven terrain (e.g., rumble strips, washboards,corrugation of road, etc.), or the like. For example, objects 106 and/orvehicles 104 operating on a gravel road may generate a larger amount ofnoise than when operating on a smooth surface. The increase in noisegenerated by the objects 106 and/or vehicles 104 (e.g., impact amount ofnoise from travel) may result in a subsequent increase in the determinedvolume and/or volume range of the warning signal.

In various examples, the one or more characteristics of the first set ofcharacteristics may be determined based on a location of the object 106in the environment 100. For example, if the object 106 is located in aroadway shared by the vehicle 104, a volume and/or volume range and/orluminosity may be higher than if the object 106 is located on thesidewalk, such as indicating an intent to enter the roadway. For anotherexample, if the object 106 is a pedestrian standing on a median betweenopposite direction traffic, a volume and/or volume range and/orluminosity may be higher than if the object 106 is located in a bikelane, proximate a curb.

In some examples, the one or more characteristics of the first set ofcharacteristics may be determined based on a detected loss of one ormore sensors 108 on the vehicle 104. For example, the vehicle computingsystem may determine that a speaker on the vehicle is not functioning atan optimal capacity. Accordingly, the vehicle computing system mayincrease a volume of the warning signal to compensate for the decreasedcapacity of the speaker. For another example, the vehicle computingsystem may determine that a light on the vehicle 104 is not functioning.Accordingly, the vehicle computing system may increase a luminosityand/or frequency of flashing of a visual warning signal to compensatefor the non-functioning light.

In various examples, the one or more characteristics of the first set ofcharacteristics may be determined based on a detection of a passenger inthe vehicle 104. In some examples, the detection of the passenger may bebased on sensor data received from one or more sensor(s) 108 of thevehicle. In some examples, the detection of the passenger may be basedon a signal received, such as from a computing device associated withthe passenger, indicating the passenger presence in the vehicle. Invarious examples, the vehicle computing system may decrease a volumeand/or volume range and/or frequencies of an audio warning signal basedon the detection of the passenger, such as, for example, to not create anegative experience for the passenger due to the emission of a loudnoise.

In various examples, the warning signal component 114 may generate(e.g., determine the first set of characteristics) and/or cause thefirst warning signal to be emitted based on a location associated withthe vehicle 104. The location may include a school zone, a constructionzone, proximity to a playground, a business district, a downtown area,or the like. In some examples, the warning signal component 114 maygenerate and/or cause the first warning signal to be emitted based on atime of day, day of the week, season, date (e.g., holiday, etc.), or thelike. In some examples, the warning signal component 114 may generateand/or cause the first warning signal to be emitted based on a speedassociated with the vehicle 104. In such examples, based on adetermination that the vehicle 104 is traveling at or below a thresholdspeed (e.g., 28 kilometers per hour, 22 miles per hour, 15 miles perhour, etc.), the warning signal component 114 may cause the firstwarning signal to be emitted.

In various examples, the computing system(s) 102 may be configured todetermine that the object(s) 106, such as object 106(1) and 106(2) arerelevant to the vehicle 104 (e.g., a potential conflict between thevehicle 104 and object 106 may exist, object 106 may potentially slowforward progress of the vehicle 104). In various examples, an objectrelevance may be determined utilizing the techniques described in U.S.patent application Ser. No. 16/389,720, filed Apr. 19, 2019, andentitled “Dynamic Object Relevance Determination,” U.S. patentapplication Ser. No. 16/417,260, filed May 20, 2019, and entitled“Object Relevance Determination,” and U.S. patent application Ser. No.16/530,515, filed Aug. 2, 2019, and entitled “Relevant ObjectDetection,” the entire contents of which are incorporated herein byreference.

In some examples, object relevance may be determined based on a distance(D) between the object 106(1) and a drivable surface 116 (e.g., aroadway, lane in which the vehicle 104 operates, etc.). In suchexamples, the object 106 may be determined to be relevant based on thedistance (D) being equal to or less than a threshold distance (e.g., 18inches, 1 foot, 4 meters, etc.). In various examples, the thresholddistance may be determined based on the classification 112 associatedwith the object 106. For example, a first threshold distance associatedwith a pedestrian may be 1 meter and a second threshold distanceassociated with a cyclist may be 5 meters.

In various examples, the object 106 may be determined to be relevantbased on an object trajectory associated therewith. In such examples,the computing system(s) 102 may be configured to determine a predictedobject trajectory (e.g., object trajectory), such as based on the sensordata. In some examples, the object trajectory may be based on a top-downrepresentation of an environment, such as by utilizing the techniquesdescribed in U.S. patent application Ser. No. 16/151,607, filed Oct. 4,2018 and entitled “Trajectory Prediction on Top-Down Scenes,” and inU.S. patent application Ser. No. 16/504,147, filed Jul. 5, 2019 andentitled “Prediction on Top-Down Scenes based on Action Data,” theentire contents of which are incorporated herein by reference. In someexamples, the predicted object trajectory may be determined using aprobabilistic heat map (e.g., discretized probability distribution),tree search methods, temporal logic formulae, and/or machine learningtechniques to predict object behavior, such as that described in U.S.patent application Ser. No. 15/807,521, filed Nov. 8, 2017, and entitled“Probabilistic Heat Maps for Behavior Prediction,” the entire contentsof which are incorporated herein by reference.

In various examples, the object 106 may be relevant to the vehicle 104based on an intersection between the object trajectory and a vehicletrajectory. In some examples, the object 106 may be relevant based onpredicted locations of the object 106 and the vehicle 104 on therespective trajectories. In some examples, the object 106 may berelevant to the vehicle 104 based on a determination that a predictedfuture object location associated with the object 106 traveling on theobject trajectory is within a threshold distance (e.g., 2 feet, 10 feet,2 meters, 4 meters, etc.) of a predicted future vehicle locationassociated with the vehicle 104 traveling on the vehicle trajectory.

In various examples, the object 106 may be relevant to the vehicle 104based on a probability of conflict (e.g., likelihood of collision)between the object 106 and the vehicle 104. The probability of conflictmay be based on a determined likelihood that the object 106 willcontinue on the object trajectory and/or alter the object trajectory toone that conflicts with the vehicle 104. In some examples, theprobability of conflict may correspond to a likelihood (e.g.,probability) of conflict between the vehicle 104 and the object 106being above a threshold level (e.g., threshold probability) of conflict.

In various examples, the vehicle computing system may determine theprobability of conflict utilizing a top down representation of theenvironment, such as that described in the U.S. patent applicationsincorporated herein above. In some examples, the vehicle computingsystem may input the top down representation of the environment into amachine learned model configured to output a heat map indicationpredicting probabilities associated with future positions of the object106 (e.g., predicting object trajectories and/or probabilitiesassociated therewith). In such examples, the vehicle computing systemmay project the movement of the vehicle 104 forward in time anddetermine a probability of conflict between an amount of overlap betweenthe heat map associated with the object 106 and future positions of thevehicle 104 as determined by the projection forward in time.

In some examples, the probability of conflict may be determined based ona classification 112 associated with the object 106. In such examples,the classification 112 associated with the object 106 may assist indetermining the likelihood that the object 106 will maintain or alter atrajectory. For example, a deer detected on a side of a roadway may beunpredictable and thus may have a high likelihood of altering atrajectory to conflict with the vehicle 104. As such, the deer may bedetermined to be an object 106 that may potentially conflict with (e.g.,is relevant to) the vehicle 104.

In some examples, based on a determination of relevance, the warningsignal component 114 of the computing system(s) 102 may generate thefirst warning signal to alert the relevant object(s) 106 in theenvironment of the vehicle 104 presence and/or operation. As discussedabove, a first set of characteristics (e.g., frequency, volume,luminosity, color, shape, motion, etc.) of the first warning signal maybe determined based on classifications 112 associated with the relevantobject(s) 106.

In various examples, the first warning signal may be emitted via one ormore emitters 118 on the vehicle 104. The emitter(s) 118 may includespeakers, lights, displays, projectors, and/or any other deviceconfigured to emit a signal. In some examples, the first warning signalmay be emitted in a plurality of directions around the vehicle (e.g.,substantially equally in front of, behind, and on the sides of thevehicle 104). In some examples, the first warning signal may be emitteduniformly in multiple directions around the vehicle 104. For example, anelectric vehicle warning sound may be emitted via speakers mounted oncorners of the vehicle 104 and configured to broadcast the first warningsignal substantially equally around the vehicle 104.

In some examples, the warning signal component 114 may be configured tocause the first warning signal to be emitted toward the relevantobject(s) 106 and/or toward detected objects 106 in the environment 100.In some examples, the first warning signal may be emitted via one ormore emitters 118 substantially facing a direction in which theobject(s) 106 (e.g., detected objects, relevant objects, etc.) aredetected. For example, objects 106 may be detected on ahead of and on aright side of the vehicle 104 (e.g., on the sidewalk adjacent to thedrivable surface 116). Based on the detection of the objects ahead ofand on the right side of the vehicle 104, the warning signal component114 may cause the first warning signal to be emitted via emitter(s) 118mounted on the front and right side of the vehicle 104. In someexamples, the first warning signal may be emitted toward the relevantobject(s) 106 and/or toward detected objects 106 in the environment 100utilizing beam steering and/or beamformed array techniques, such as thatdescribed in U.S. Pat. No. 9,878,664, issued May 4, 2017 and entitled“Method for Robotic Vehicle Communication with an External Environmentvia Acoustic Beam Forming,” the entire contents of which areincorporated herein by reference.

In some examples, the warning signal component 114 may be configured tocontinually and/or periodically (e.g., every 0.5 seconds, 3.0 seconds,etc.) modify the first warning signal to generate the second warningsignal, and so on. In some examples, the modification to the warningsignals may be based at least in part on additional sensor dataprocessed by the computing system(s) 102. For example, the warningsignal component 114 may cause the first warning signal to be emitted ata first time, at least one characteristic thereof being determined basedon a baseline noise level in the environment. The vehicle computingsystem may determine an increase in the baseline noise level at a secondtime and the warning signal component 114 may generate a second warningsignal to be emitted at a higher volume.

In various examples, the warning signal component 114 may be configuredto determine an object reaction to the first warning signal. In suchexamples, the warning signal component 114 may be configured todetermine a real-time object reaction to a warning signal. In someexamples, the warning signal component 114 may receive processed sensordata from the perception component 110, such as that associated with anobject reaction of the object(s) 106 in the environment 100. The objectreaction may include a change (or lack thereof) in the object trajectory(e.g., speed increase, speed decrease, direction of travel away from thevehicle, etc.), a movement of the head and/or shoulders of the object, agesture (e.g., a wave, etc.), a foot placement of the object, apositional adjustment to an item the object holds (e.g., adjusting aposition of an electronic device, book, magazine, or other item), and/orany other movement indicative of an object reacting the first warningsignal.

The warning signal component 114 may compare the object reaction to anexpected reaction 120. The expected reaction may be based on thecharacteristic(s) of the first warning signal (e.g., volume, frequency,luminosity, color, motion (e.g., animated motion, light sequencing,etc.), shape of the signal, etc.) and/or data associated with theobject(s) 106 (e.g., object attribute (e.g., classification 112,position (e.g., facing/moving toward the vehicle 104, facing/moving awayfrom the vehicle 104, etc.), distance (D) from the vehicle 104, objecttrajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object 106 (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.). In some examples, the warning signal component 114may access a database 122 including a plurality of expected reactions todetermine the expected reaction 120 associated with the first warningsignal. In such examples, the expected reactions 120 in the database 122may be stored based at least in part on the data associated with theobject 106 and/or characteristic(s) of the first warning signal. Forexample, the database 122 may include an expected reaction 120(1) of theobject 106(1), a pedestrian viewing an electronic device, to includelowering the electronic device the object 106(1) is viewing and/ormoving the head and/or shoulders toward the vehicle 104 (e.g., emissionof the first warning signal). For another example, the database 122 mayinclude an expected reaction 120(2) of the object 106(2), a cyclist, toinclude a change in object trajectory (e.g., increase in speed, decreasein speed, change of direction of travel, etc.) and/or a head movement.In the illustrative example, the database 122 may be located on theautonomous vehicle 104 separate from the computing system(s) 102. Insuch examples, the database 122 may be accessible to the computingsystem(s) 102 via a wired and/or wireless connection. In some examples,the database 122 may be remote from the computing system(s) 102, such asthat stored on a remote computing system and accessible via the wirelessconnection. In yet other examples, the database 122 may be located onthe computing system(s) 102.

As illustrated with respect to object 106(2), the expected reaction120(2) may include a change in trajectory associated with the object106(2). The change in the trajectory may include a modification to thespeed (e.g., speed up, slow down, change speed a threshold amount, etc.)and/or direction the object 106(2) travels. In various examples, thecomputing system 102 may determine an updated predicted objecttrajectory based on additional sensor data from the sensors at a timeafter emitting the first warning signal. In some examples, the updatedpredicted object trajectory may be determined utilizing the top-downrepresentation of the environment and/or heat maps associated therewith,such as that described in the U.S. patent applications incorporatedherein by reference above. In various examples, the computing system(s)102 may determine a modification to the object trajectory (e.g.,difference between the predicted object trajectory and the updatedpredicted object trajectory determined after emitting the first warningsignal). In some examples, the expected reaction may be based on themodification. For example, the expected reaction may include an objectslowing a forward speed or changing a direction of travel (e.g., from anintersecting trajectory to a parallel trajectory with the vehicle 104).The computing system(s) 102 may compare the modification and/or updatedobject trajectory to the expected reaction 120(2) to determine whetherthe object 106(2) reacts in accordance with the expected reaction.

In some examples, the computing system(s) 102 may determine that thefirst warning signal was successful in alerting the object 106 based ona determination that the object reacts (e.g., modifies behavior) withina threshold time (e.g., 1 second, 2 seconds, etc.) of emitting the firstwarning signal. In some examples, the computing system(s) 102 may storethe response associated with the reaction in the database 122. Invarious examples, despite detecting an unexpected reaction (e.g., notthe expected reaction), the computing system(s) 102 may continue toiteratively modify warning signals until the expected reaction isdetected. In such examples, the computing system(s) may store dataassociated with each iteration of the warning signal and correspondingreactions (expected and/or unexpected) in the database 122.

In various examples, the vehicle computing system may determine anexpected reaction 120 utilizing machine learning techniques. As will bediscussed in greater detail below with regard to FIG. 4, in someexamples, the computing system(s) 102 may include a reaction trainingcomponent configured to train a model utilizing machine learningtechniques to determine an expected reaction 120 to a warning signal. Insuch examples, the model may be trained with training data including aplurality of warning signals and detected reactions thereto.

Based on the comparison between the object reaction (e.g., actualreaction, real-time reaction, etc.) and the expected reaction 120, thewarning signal component 114 may determine whether the object 106reacted as expected to the first warning signal (e.g., whether asubstantial match exists between the object reaction and the expectedreaction 120). In various examples, a determination of a substantialmatch between the object reaction and the expected reaction 120 mayinclude a match of a threshold number of actions (e.g., one matchingactions, two matching actions, etc.), a threshold percentage of actions(e.g., 80%, 55%, etc.), or the like. In some examples, the substantialmatch may be determined based on a threshold match and/or thresholddifference between the object reaction and the expected reaction 120.The actions may include trajectory modifications (e.g., increase inspeed, decrease in speed, change in direction of travel, etc.), bodymovements (e.g., foot placement, head rotation, shoulder movement,etc.), gestures, or the like. For example, an expected reaction 120(1)to the first warning signal may include a head and/or shoulder movementand a positional adjustment to an electronic device the object 106(1)holds. The actual object reaction may include a head movement toward thevehicle 104. Based on a match of at least the head movement, the warningsignal component 114 may determine that the object reaction and theexpected reaction 120 substantially match. For another example, anexpected reaction 120(2) to the first warning signal may include a headmovement, a modification to an object trajectory, and/or a magnitude ofthe modification to the object trajectory. The warning signal component114 may receive an indication, such as from a prediction component, thata speed associated with the object trajectory has decreased by 5 milesper hour. Based in part on the object trajectory modification andmagnitude thereof, the warning signal component may determine that theobject reaction matches the expected reaction 120(2) at 85%, above a 75%threshold percentage of actions, and that the object reactionsubstantially matches the expected reaction 120(2).

Responsive to a determination that the object reaction substantiallymatches the expected reaction 120, the vehicle computing system maystore the encounter (e.g., data associated with first warning signal andthe object reaction) in the database 122. In some examples, the database122 may be used for future object reaction comparisons, such as toincrease a confidence in an object reaction to the first warning signal,to train the machine learned model, or the like.

Responsive to a determination that the object reaction did notsubstantially match (e.g., less than the threshold number of actions,percentage match, etc.), the warning signal component 114 may determinethat the object 106 did not react according to an expected reaction 120.In such examples, the warning signal component 114 may determine thatthe object 106 remains unaware of the vehicle 104 operation and/orpresence of the vehicle 104 in the environment 100. Based on adetermination that the object 106 did not react according to theexpected reaction 120, the warning signal component 114 may generate asecond warning signal. The second signal may include an audio and/orvisual signal. The second warning signal may include a signal of a sameor a different modalit(ies) (e.g., light, sound, etc.) from the firstwarning signal. For example, the first warning signal may include asound emission and the second warning signal may include a lightemission. For another example, the first warning signal may include asound emission and the second warning signal may include a soundemission.

In various examples, the warning signal component 114 may determine asecond set of characteristics (e.g., frequency, volume, luminosity,color, motion (e.g., animated motion, light sequencing, etc.), shape ofthe signal, etc.) of the second warning signal. In some examples, thesecond set of characteristics may include a pre-determined modificationto one or more of the characteristics of the first warning signal. Insuch examples, the warning signal component 114 may modify one or moreof the frequenc(ies), volume, and/or luminosity of the first warningsignal to generate the second warning signal. For example, a volumeassociated with a second warning signal may include a 10 decibelincrease from the first warning signal.

In various examples, the second set of characteristics may be determineddynamically, such as based on real-time conditions in the environment100, as discussed above. In some examples, the warning signal component114 may process the real-time considerations and a failure to react tothe first warning signal and may determine the second set ofcharacteristics associated with the second warning signal. In someexamples, the warning signal component 114 may access the database 122to determine the second set of characteristics associated with thesecond signal. In some examples, the second set of characteristics maybe stored in the database 122 based on the real-time considerations. Invarious examples, the second set of characteristics may be determinedutilizing machine learning techniques. In some examples, the warningsignal component 114 may input the real-time considerations and thefirst set of characteristics with an indication that the first warningsignal was unsuccessful into a machine learned model configured tooutput the second set of characteristics. In such examples, the warningsignal component 114 may generate the second warning signal according tothe output second set of characteristics.

In various examples, the second set of characteristics may be based inpart on an escalation of urgency, such as from a low urgency to mediumor high urgency, an increase in a likelihood and/or probability ofconflict, such as from a medium probability to a high probability, orthe like. For example, the warning signal component 114 may cause afirst warning signal to be emitted at a first frequency and first volumeto alert an object 106, such as object 106(1), that is within athreshold distance of the drivable surface 116 of the vehicle operation.Based on a determination that the object 106(1) did not react accordingto an expected reaction 120(1), and that the vehicle 104 has movedcloser to a location associated with the object 106(1), the vehiclecomputing system may determine that the urgency associated with thenotification and/or a probability of conflict has increased.Accordingly, the warning signal component 114 may determine to modify afrequency and/or increase a volume of the first warning signal togenerate the second warning signal.

In various examples, the second set of characteristics may include apre-determined modification to the first set of characteristics. In suchexamples, one or more of the first set of characteristics may bemodified by a pre-defined amount to determine the second set ofcharacteristics. In some examples, the pre-defined modifications may bestored in the database 122, such as in association with the firstwarning signal and/or characteristics thereof, the data associated withthe object 106, or the like. For example, a volume associated with thesecond warning signal may include a 10 decibel increase over the firstwarning signal. For another example, the second warning signal mayinclude a 100 lumen increase over the first warning signal.

In various examples, the warning signal component 114 may compare asecond object reaction to a second expected reaction 120 associated withthe second warning signal. Responsive to a determination that the secondobject reaction substantially matches the second expected reaction 120,the warning signal component 114 may store data associated with theobject reaction and/or the second warning signal in the database 122. Asdiscussed above, in some examples, the database 122 may be used forfuture object reaction comparisons, such as to increase a confidence ina reaction to a warning signal, to train the machine learned model, orthe like.

Responsive to a determination that the second object reaction does notsubstantially match the second expected reaction 120, the warning signalcomponent 114 may generate and cause a third warning signal to beemitted, the third warning signal being different from the first warningsignal and the second warning signal (e.g., different modality,different characteristics, etc.). For example, based on a determinationthat the object 106 did not respond to a first audio warning signalemitted at 50 decibels and a second audio warning signal emitted at 70decibels, the warning signal component 114 may determine to emit avisual warning signal as the third warning signal. Accordingly, thewarning signal component 114 may determine a color and luminosity of thevisual warning signal (e.g., third set of characteristics associatedwith the third warning signal). The warning signal component 114 mayutilize similar or the same techniques as those described above withregard to determining a third set of characteristics associated with thethird warning signal.

In various examples, the vehicle computing system may continue to modifyemitted warning signals until an object reaction substantially matchesan expected reaction 120 to the warning signal. In various examples, thewarning signal component 114 may continue to modify emitted warningsignals based on a determination that the object 106 remains relevant tothe vehicle 104. In such examples, a modified warning signal may beemitted based on a determination that the object 106 is relevant to thevehicle 104. In some examples, the warning signal component 114 may beconfigured to continually and/or periodically (e.g., every 0.1 seconds,1.0 seconds, prior to generating a modified warning signal, etc.)determine whether the object 106 is relevant to the vehicle 104. In someexamples, the warning signal component 114 may receive an indication ofobject 106 relevance from another component of the computing system(s)102. In various examples, the indication of object 106 relevance may bereceived responsive to a query of relevance sent by the warning signalcomponent 114. In such examples, the warning signal component 114 maysend the query of relevance prior to generating a modified warningsignal, such as to verify object 106 relevance prior to expendingcomputing resources to generate the modified warning signal. In suchexamples, the techniques described herein may improve the functioning ofthe computing system(s) 102, at least by making available additionalcomputing resources (processing power, memory, etc.) to other functionsof the computing system(s) 102 based on a determination of object 106irrelevance.

In various examples, based on a determination that the object 106reacted in accordance with an expected reaction 120 and/or that theobject 106 is irrelevant to the vehicle 104, the warning signalcomponent 114 may cease emitting warning signals. In some examples,based on the determination that the object 106 reacted in accordancewith the expected reaction 120 and/or the object 106 is irrelevant tothe vehicle 104, the warning signal component 114 may cause the firstwarning signal to be emitted. In such examples, the first warning signalmay include a baseline warning signal emitted to alert nearby objects ofthe vehicle 104 presence and/or operation. For example, and as discussedabove, the baseline warning signal may include an electric vehiclewarning sound, such as that required by law and/or regulation. Foranother example, the baseline warning signal may include a sound and/orlight emitted based on a location associated with the vehicle 104 and/orreal-time conditions such as those described above.

FIG. 2 is an illustration of a process 200 for modifying a warningsignal 202 emitted by a vehicle 104 based at least in part on an objectreaction 204 of an object 106 to the emitted warning signal 202.

At operation 206, the process may include emitting a first warningsignal 202(1) in an environment, such as environment 100. The firstwarning signal 202(1) may include an audio and/or a visual signal. Thefirst warning signal 202(1) may include a first set of characteristics(e.g., frequency, volume, luminosity, color, motion (e.g., animatedmotion, light sequencing, etc.), shape of the signal, etc.). In someexamples, the first set of characteristics associated with the firstwarning signal 202(1) may be pre-defined, such as based on a baselinesound emitted from the vehicle 104 to alert objects 106 of the presenceand/or operation of the vehicle 104. For example, the first warningsignal 202(1) may include an electric vehicle warning sound with afrequency of 528 Hertz emitted at 100 decibels.

In various examples, a vehicle computing system associated with thevehicle 104 may determine the first set of characteristics based on oneor more real-time conditions in the environment. As discussed above, thereal-time conditions may include data associated with the object 106(e.g., object attribute (e.g., classification, position (e.g.,facing/moving toward the vehicle, facing/moving away from the vehicle,etc.), distance from the vehicle, trajectory, etc.), object activity(e.g., walking, running, riding a scooter, (e.g., a particular activityimplied by an object trajectory, such as based on speed, etc.), readinga book, talking on a phone, viewing data on an electronic device,interacting with another vehicle, interacting with another object (e.g.,talking to another person, looking into a stroller, etc.), eating,drinking, operating a sensory impairment device (e.g., cane, hearingaid, etc.), listening to headphones, etc.), environmental factors (e.g.,noise level in the environment, amount of traffic, road conditions,etc.), weather conditions (e.g., rain, snow, hail, wind, etc.),vehicular considerations (e.g., speed, passengers in the vehicle 104,etc.), and the like. For example, the vehicle computing system maydetermine that a detected object 106 is a pedestrian walking in the raintoward the vehicle 104 and within a threshold distance of the vehicle104. Due to the object trajectory, the object location within thethreshold distance of the vehicle 104, and the rain, the vehiclecomputing system may determine to emit a 1000 Hertz signal at 100decibels.

In the illustrative example, the first warning signal 202(1) may beemitted on a side of the vehicle 104 associated with the detected object106. In such an example, the first warning signal 202(1) may beconfigured to alert objects 106 in a bike lane, on a sidewalk, and/orother area in which the vehicle computing system detects objects 106and/or reasonably expects objects to operate (e.g., lawful areas ofoperation, typical areas of operation, etc.). In some examples, thefirst warning signal 202(1) may be emitted around the vehicle, such astoward the front, rear, right side and left side. In some examples, thefirst warning signal 202(1) may include an acoustic beamformed signaland/or light signal directed at the object 106.

At operation 208 the vehicle computing system may compare an objectreaction 204 to an expected reaction 120 to the first warning signal202(1). As discussed above, the vehicle computing system may determinethe object reaction 204 based on sensor data from one or more sensors.The sensors may include sensors mounted on the vehicle, mounted on othervehicles, and/or mounted in the environment. The object reaction 204 mayinclude a real-time object reaction to the first warning signal 202(1),such as a change or lack thereof to an object trajectory, position,and/or movement of the object 106. In the illustrative example, theobject reaction 204 includes the substantial lack of movement. Forexample, a position of the object 106 (e.g., head, shoulders, arms,legs, etc.) remains substantially the same and the object 106 continuesto hold an electronic device in substantially the same position.

The expected reaction 120 may be based on the characteristic(s) of thefirst warning signal 202(1) (e.g., volume, frequency, luminosity, color,motion (e.g., animated motion, light sequencing, etc.), shape of thesignal, etc.) and/or data associated with the object 106 (e.g., objectattribute (e.g., classification, position (e.g., facing/moving towardthe vehicle, facing/moving away from the vehicle, etc.), distance fromthe vehicle, trajectory, etc.), object activity (e.g., walking, running,riding a scooter, (e.g., a particular activity implied by an objecttrajectory, such as based on speed, etc.), reading a book, talking on aphone, viewing data on an electronic device, interacting with anothervehicle, interacting with another object (e.g., talking to anotherperson, looking into a stroller, etc.), eating, drinking, operating asensory impairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.). In some examples, the vehicle computing system mayaccess a database including a plurality of expected reactions todetermine the expected reaction 120 associated with the first warningsignal 202(1). In such examples, the expected reactions 120 in thedatabase may be stored based at least in part on the data associatedwith the object 106 and/or characteristic(s) of the first warning signal202(1). In various examples, the vehicle computing system may determinethe expected reaction 120 utilizing machine learning techniques. In theillustrative example, the expected reaction 120 includes a head movement(e.g., rotation) toward the vehicle 104 and a movement of an electronicdevice from an elevated position (e.g., in front of the object's head)to a lower position.

At operation 210, the process includes emitting a second warning signalbased on the object reaction 204 being substantially different from theexpected reaction 120. In various examples, the vehicle computing systemmay determine that the object 106 did not process the first warningsignal 202(1) (e.g., object 106 did not hear and/or see the firstwarning signal 202(1)) based on the substantial difference between theobject reaction 204 and the expected reaction 120.

A determination of substantial difference may be based on one or moreactions associated with the object reaction 204 differing from one ormore actions of the expected reaction 120. The actions may includetrajectory modifications (e.g., increase in speed, decrease in speed,change in direction of travel, etc.), body movements (e.g., footplacement, head rotation, shoulder movement, etc.), gestures, or thelike. In some examples, the object reaction 204 may be determined to besubstantially different from the expected reaction 120 based on adetermination that a threshold number of actions and/or thresholdpercentage of actions are different (e.g., threshold difference). Forexample, an expected reaction may include a foot placement, a headmovement and a shoulder movement. Based on a determination that anobject reaction only includes a foot placement, the vehicle computingsystem may determine that a threshold number of actions (two) is not metand, therefore, the object reaction is substantially different from theexpected reaction.

The second warning signal 202(2) may include a signal of the same or adifferent modality from the first warning signal 202(1). The secondwarning signal 202(2) may include a second set of characteristics. Invarious examples, the vehicle computing system may determine the secondset of characteristics based on a pre-defined adjustment to the firstset of characteristics. In such an example, the vehicle computing systemmay modify a frequency, volume, luminosity, color, motion, and/or shapeof the first warning signal 202(1) based on the pre-defined adjustment.For example, a first warning signal 202(1) may include an audio signalof a first frequency emitted at 65 decibels. Based on a determinationthat the object 106 did not react according to an expected reaction, thevehicle computing system may increase the volume 15 decibels and emit asecond warning signal 202(2) at 80 decibels.

In the illustrative example, the vehicle computing system may cause thesecond warning signal 202(2) to be emitted on a same side of the vehicle104 as the first warning signal 202(1) (e.g., on a side of the vehicle104 associated with the detected object 106). In other examples, thevehicle computing system may cause the second warning signal 202(2) tobe directed at the object 106, such as in the acoustic beamformed signaland/or light signal. In yet other examples, the second warning signal202(2) may be emitted around the vehicle, such as toward the front,rear, right side, and left side of the vehicle 104.

At operation 212, the process includes emitting different warningsignals 202 until the object reaction 204 matches (e.g., substantiallymatches) the expected reaction 120 or the object 106 is no longerrelevant to the vehicle 104. In various examples, the vehicle computingsystem may be configured to continuously modify warning signals based onreal-time object reactions in order to optimize the safe operation ofthe vehicle 104 in the environment.

In various examples, the vehicle computing system may generate differentwarning signals 202 of a same modality and/or different modalities. Insome examples, the vehicle computing system may emit a pre-determinednumber of warning signals 202 in a first modality and may change to asecond modality. In such examples, the vehicle computing system maydetermine that the first modality is ineffective in alerting the object106 of the vehicle 104 presence and/or operation. For example, thevehicle computing system may emit three audio warning signals 202 andmay determine, based on a substantial difference between the objectreactions 204 to the audio signals and expected reactions 120 thereto,that the audio signals are ineffective, such as due to the object 106being hard of hearing, listening to loud music, or the like. The vehiclecomputing system may modify the fourth warning signal 202 (andsubsequent warning signals 202) to visual warning signals 202, such asflashing lights of various colors, motions (e.g., sequencing), shapes,and/or intensity.

In various examples, the vehicle computing system may be configured tocontinuously and/or periodically (e.g., every 0.2 seconds, 0.5 seconds,prior to generating a modified warning signal, etc.) determine whetherthe object 106 is relevant to the vehicle 104. In some examples, priorto generating a subsequent warning signal 202 (such as after adetermination that the object reaction 204 is substantially differentfrom the expected reaction 120), the vehicle computing device maydetermine object relevance to the vehicle 104. The object 106 may berelevant to the vehicle 104 based on an intersection between an objecttrajectory and a vehicle trajectory. The object trajectory may bedetermined utilizing the techniques described above and in the patentapplications incorporated by reference above. In some examples, theobject 106 may be relevant based on predicted locations of the object106 and the vehicle 104 on the respective trajectories.

In some examples, the object 106 may be relevant to the vehicle 104based on a determination that a predicted future object locationassociated with the object 106 traveling on the object trajectory iswithin a threshold distance (e.g., 4 feet, 12 feet, 1 meters, 3 meters,etc.) of a predicted future vehicle location associated with the vehicle104 traveling on the vehicle trajectory. In various examples, the object106 may be determined to be relevant to the vehicle 104 based on alocation of the object 106 being in front of the vehicle 104 (e.g.,ahead of the vehicle 104 traveling in a direction) and a distancebetween the object 106 and a drivable surface (e.g., a road) on whichthe vehicle 104 is traveling on the trajectory (e.g., distance from theobject to a path of the vehicle 104). In such examples, the object 106may be relevant based on a determination that the distance is equal toor less than a threshold distance.

In various examples, based on a determination that the object 106reacted in accordance with an expected reaction 120 and/or that theobject 106 is irrelevant to the vehicle 104, the vehicle computingdevice may cease emitting warning signals 202. In some examples, basedon the determination that the object 106 reacted in accordance with theexpected reaction 120 and/or the object 106 is irrelevant to the vehicle104, the warning signal component 114 may cause the first warning signal202(1) to be emitted. In such examples, the first warning signal mayinclude a baseline warning signal 202 emitted to alert nearby objects106 of the vehicle 104 presence and/or operation. For example, thebaseline warning signal may include a sound and/or light emitted basedon a location associated with the vehicle 104 and/or real-timeconditions such as environmental factors, weather conditions, vehicularconsiderations, data associated with the object 106, and the like.

FIG. 3 is an illustration of an environment 300 in which a vehicle 302,such as vehicle 104 emits signals 304 and 306 based on a determinationthat an object 308(1) is blocking a vehicle path 310 of the vehicle 302.The signals may include a warning signal 304, such as warning signal 202to alert the object 106 of the presence and/or operation of the vehicle302, and a routing signal 306 to indicate a potential object path forthe object 106. A vehicle computing system associated with the vehiclemay be configured to detect objects 308, such as object(s) 106, in theenvironment based at least in part on sensor data received from one ormore sensors of the vehicle and/or one or more remote sensors (e.g.,sensors associated with other vehicles, sensors mounted in theenvironment 300, etc.).

In various examples, a vehicle computing system, such as computingsystem(s) 102, may be configured to determine that an object 308 (e.g.,blocking object 308(1)) is blocking the vehicle path 310 (path 310)associated with vehicle 302 travel through the environment 300. In someexamples, the vehicle path 310 may include a path of the vehicle 302from a current location 312 to a destination. In some examples, thevehicle path 310 may include a drivable surface (e.g., drivable area)associated with the vehicle 302 travel to the destination. In someexamples, the drivable surface may include a width of the vehicle 302and/or a safety margin on either side of the vehicle 302. In someexamples, the drivable surface may include the width of a lane 314 inwhich the vehicle 302 is traveling.

In some examples, the vehicle computing system may determine that theblocking object 308(1) is blocking the vehicle path 310 based on adetermination that an object location 316 associated with the blockingobject(s) 308(1) is at least partially within the drivable area and/orthe vehicle path 310. In various examples, the vehicle computing systemmay determine that the blocking object 308(1) is blocking the vehiclepath 310 based on a determination that the vehicle 302 is not able toproceed around the blocking object 308(1) in the lane 314. In theillustrative example, the blocking object 308(1) is stopped across thevehicle path 310 and is blocking the lane 314. In such an example, thevehicle 302 may be unable to proceed along the vehicle path 310 in thelane 314 (or an adjacent lane 318). In other examples, the blockingobject 308(1) may be blocking less of the lane 314 (e.g., smallerpercentage of the blocking object 308(1) blocking the vehicle path 310);however, the vehicle computing system may determine that the object 308is a blocking object 308(1) based on a determination that the vehicle302 is unable to circumnavigate the blocking object 308(1) whileremaining within the confines of the lane 314.

In various examples, the vehicle computing system may be configured toidentify an area 320 into which the blocking object 308(1) may move. Insome examples, the area 320 may include a location that is not in thevehicle path 310, the lane 314, and/or the adjacent lane 318. In suchexamples, the area 320 may include a location to which the blockingobject 308(1) may move to no longer block progress of the vehicle 302and/or other vehicles/objects 308 traveling in the lane 314 and/or theadjacent lane 318.

In some examples, the area 320 may include a location that the operatorof the blocking object 308(1) may be unable to view, such as based on aviewing path being blocked by another object, such as object 308(2). Forexample, the blocking object 308(1) may be turning left in anintersection 322 in front of the vehicle 302. The blocking object 308(1)may be operating in the left-hand lane behind the object 308(2). Due tothe object location 316 and position (e.g., orientation partiallythrough the left turn), the operator of the blocking object 308(1) maybe unable to see whether the area 320 in the right-hand lane is clear ofobjects 308.

In various examples, the vehicle computing system may cause a routingsignal 306 to be emitted from an emitter, such as emitter 118. In someexamples, the routing signal 306 may include an indication of an objectpath (route) out of the vehicle path 310. In some examples, the routingsignal 306 may indicate to the operator of the blocking object 308(1)that the area 320 exists and is clear. In the illustrative example, therouting signal 306 includes an arrow pointing to the area 320, therouting signal 306 projected (e.g., displayed) on a surface of the roadsuch that the operator of the blocking object 308(1) may view therouting signal 306 from the object location 316. In another example, therouting signal 306 may include a holographic image providing anindication of the area 320 into which the blocking object 308(1) maymove. Though depicted in FIG. 3 as an arrow, this is merely forillustrative purposes and other designs, shapes, symbols, and the likeare contemplated here. For example, the routing signal 306 may include aflashing sequence of lights configured to indicate a route, such as anapproach lighting system.

In various examples, the vehicle computing system may emit the routingsignal 306 and may determine that the operator of the blocking object308(1) is not reacting according to an expected reaction (e.g., operatoris not moving the blocking object 308(1) toward the area 320. In someexamples, based on a determination that the operator of the blockingobject 308(1) is not reacting according to the expected reaction to therouting signal 306, the vehicle computing system may cause the warningsignal 304 to be emitted, such as to alert the operator of the blockingobject 308(1) of the routing signal 306 and/or the vehicle presenceand/or operation. In some examples, the vehicle computing system maycause the warning signal 304 to be emitted to get the attention of theoperator of the blocking object 308(1) prior to emitting the routingsignal 306. The warning signal 304 may include an audio and/or a visualsignal with a first set of characteristics (e.g., frequency, volume,luminosity, color, motion (e.g., animated motion, light sequencing,etc.), shape of the signal, etc.).

As discussed above, the vehicle computing system may be configured todetect a reaction of the operator of the blocking object 308(1) (e.g.,operator reaction) to the warning signal 304. The vehicle computingsystem may detect the operator reaction based on sensor data collectedfrom the sensors associated with the vehicle and/or the remotesensor(s). Due in part to limited visibility of vehicular operatorsprovided by sensor data, the operator reaction may include a bodymovement, such as a head movement, shoulder movement, hand gesture(e.g., wave, etc.), and the like. As discussed above, the vehiclecomputing system may be configured to compare the operator reaction toan expected reaction, such as expected reaction 120. Additionally, inexamples in which the vehicle computing system causes the routing signal306 to be emitted concurrently with or prior to the warning signal 304,the expected reaction may include movement of the blocking object 308(1)toward the area 320. Based on the comparison, the vehicle computingsystem may determine whether the operator of the blocking object 308(1)is aware of the presence and/or operation of the vehicle 302 and/or therouting signal 306.

Based on a determination that the operator of the blocking object 308(1)is not aware of the presence and/or operation of the vehicle 302 (e.g.,operator reaction did not substantially match the expected reaction)and/or the routing signal 306, the vehicle computing system may modifythe warning signal 304. The modification may include a change inmodality (e.g., audio signal to visual signal), frequency, volume,luminosity, color, motion, shape, or the like. In various examples, thevehicle computing system may emit a modified warning signal 304 to alertthe operator of the blocking object 308(1) of the presence and/oroperation of the vehicle 302 and/or the routing signal 306. As discussedabove, the vehicle computing system may be configured to continuallymodify the warning signal 304 until the operator reaction substantiallymatches the expected reaction or the vehicle computing system determinesthat the blocking object 308(1) is irrelevant to the vehicle 302 (e.g.,no longer blocking the path 310).

Though FIG. 3 is described with regard to a blocking object 308(1), thisis not intended to be limiting, and the vehicle computing system may beconfigured to generate and emit the routing signal 306 for other(non-blocking) objects 308. For example, the vehicle computing systemmay detect an object 308 within a threshold distance of the vehicle 302.The vehicle computing system may determine, based on the object 308being within the threshold distance and to maximize operational safety,to slow a forward speed of the vehicle 302 if the object 308 maintains afirst location inside the threshold distance. The vehicle computingsystem may determine that, if the object moves to a second locationoutside the threshold location, that the vehicle 302 may not need toslow for optimal operational safety. The vehicle computing system maythus generate a routing signal 306 to indicate to the object 308 thesecond location for the object 308 to move so as to not impede forwardprogress of the vehicle 302.

FIG. 4 is a block diagram of an example system 400 for implementing thetechniques described herein. In at least one example, the system 400 mayinclude a vehicle 402, such as vehicle 104.

The vehicle 402 may include one or more vehicle computing devices 404(e.g., vehicle computing system), such as computing system(s) 102, oneor more sensor systems 406, such as sensor(s) 108, one or more emitters408, such as emitter(s) 118, one or more communication connections 410,at least one direct connection 412, and one or more drive systems 414.

The vehicle computing device(s) 404 may include one or more processors416 and memory 418 communicatively coupled with the one or moreprocessors 416. In the illustrated example, the vehicle 402 is anautonomous vehicle; however, the vehicle 402 could be any other type ofvehicle, such as a semi-autonomous vehicle, or any other system havingat least an image capture device (e.g., a camera enabled smartphone). Inthe illustrated example, the memory 418 of the vehicle computingdevice(s) 404 stores a localization component 420, a perceptioncomponent 422, a planning component 424, one or more system controllers426, and a warning signal component 428 including a signal emissioncomponent 430, a reaction determination component 432, a machinelearning component 434, a reaction database 436, and an object routingdetermination component 438. Though depicted in FIG. 4 as residing inthe memory 418 for illustrative purposes, it is contemplated that thelocalization component 420, a perception component 422, a planningcomponent 424, one or more system controllers 426, and a warning signalcomponent 428 (and/or the components and/or database illustratedtherein) may additionally, or alternatively, be accessible to thevehicle 402 (e.g., stored on, or otherwise accessible by, memory remotefrom the vehicle 402, such as, for example, on memory 440 of one or more(remote) computing devices 442).

In at least one example, the localization component 420 may includefunctionality to receive data from the sensor system(s) 406 to determinea position and/or orientation of the vehicle 402 (e.g., one or more ofan x-, y-, z-position, roll, pitch, or yaw). For example, thelocalization component 420 may include and/or request/receive one ormore map(s) of an environment and may continuously determine a locationand/or orientation of the autonomous vehicle within the map(s). For thepurpose of this discussion, a map may be any number of data structuresmodeled in two dimensions, three dimensions, or N-dimensions that arecapable of providing information about an environment, such as, but notlimited to, topologies (such as intersections), streets, mountainranges, roads, terrain, and the environment in general. In someinstances, a map may include, but is not limited to: texture information(e.g., color information (e.g., RGB color information, Lab colorinformation, HSV/HSL color information), and the like), intensityinformation (e.g., lidar information, radar information, and the like);spatial information (e.g., image data projected onto a mesh, individual“surfels” (e.g., polygons associated with individual color and/orintensity)), reflectivity information (e.g., specularity information,retroreflectivity information, BRDF information, BSSRDF information, andthe like). In at least one example, a map may include athree-dimensional mesh of the environment. In some examples, the vehicle402 may be controlled based at least in part on the map(s). That is, themap(s) may be additionally used in connection with the perceptioncomponent 422 and/or the planning component 424 to determine a locationof the vehicle 402, detect objects in an environment, and/or generateroutes and/or trajectories to navigate within an environment.

In some examples, the one or more maps may be stored on a remotecomputing device(s) (such as the computing device(s) 442) accessible vianetwork(s) 444. In some examples, multiple maps may be stored based on,for example, a characteristic (e.g., type of entity, time of day, day ofweek, season of the year, etc.). Storing multiple maps may have similarmemory requirements but increase the speed at which data in a map may beaccessed.

In various examples, the localization component 420 may be configured toutilize SLAM (simultaneous localization and mapping), CLAMS(calibration, localization and mapping, simultaneously), relative SLAM,bundle adjustment, non-linear least squares optimization, or the like toreceive image data, lidar data, radar data, IMU data, GPS data, wheelencoder data, and the like to accurately determine a location of thevehicle 402. In some instances, the localization component 420 mayprovide data to various components of the vehicle 402 to determine aninitial position of an autonomous vehicle 402 for determining alikelihood (e.g., probability) of conflict with an object, such aswhether the object is relevant to the vehicle 402, as discussed herein.

In some examples, the perception component 422 may include functionalityto perform object detection, segmentation, and/or classification. Insome examples, the perception component 422 may provide processed sensordata that indicates a presence of an object (e.g., entity, dynamicobject) that is proximate to the vehicle 402 and/or a classification ofthe object as an object type (e.g., car, pedestrian, cyclist, dog, cat,deer, unknown, etc.). In some examples, the perception component 422 mayprovide processed sensor data that indicates a presence of a stationaryentity that is proximate to the vehicle 402 and/or a classification ofthe stationary entity as a type (e.g., building, tree, road surface,curb, sidewalk, unknown, etc.). In additional or alternative examples,the perception component 422 may provide processed sensor data thatindicates one or more characteristics associated with a detected object(e.g., a tracked object) and/or the environment in which the object ispositioned. In some examples, characteristics associated with an objectmay include, but are not limited to, an x-position (global and/or localposition), a y-position (global and/or local position), a z-position(global and/or local position), an orientation (e.g., a roll, pitch,yaw), an object type (e.g., a classification), a velocity of the object,an acceleration of the object, an extent of the object (size), etc.Characteristics associated with the environment may include, but are notlimited to, a presence of another object in the environment, a state ofanother object in the environment, a time of day, a day of a week, aseason, a weather condition (e.g., rain, sleet, hail, snow, temperature,humidity, etc.), an indication of darkness/light, etc.

In general, the planning component 424 may determine a path for thevehicle 402 to follow to traverse through an environment. For example,the planning component 424 may determine various routes and trajectoriesand various levels of detail. For example, the planning component 424may determine a route to travel from a first location (e.g., a currentlocation) to a second location (e.g., a target location). For thepurpose of this discussion, a route may include a sequence of waypointsfor travelling between two locations. As non-limiting examples,waypoints include streets, intersections, global positioning system(GPS) coordinates, etc. Further, the planning component 424 may generatean instruction for guiding the vehicle 402 along at least a portion ofthe route from the first location to the second location. In at leastone example, the planning component 424 may determine how to guide thevehicle 402 from a first waypoint in the sequence of waypoints to asecond waypoint in the sequence of waypoints. In some examples, theinstruction may be a trajectory, or a portion of a trajectory. In someexamples, multiple trajectories may be substantially simultaneouslygenerated (e.g., within technical tolerances) in accordance with areceding horizon technique, wherein one of the multiple trajectories isselected for the vehicle 402 to navigate.

In some examples, the planning component 424 may include a predictioncomponent to generate predicted trajectories of objects in anenvironment. For example, a prediction component may generate one ormore predicted trajectories for objects within a threshold distance fromthe vehicle 402. In some examples, a prediction component may measure atrace of an object and generate a trajectory for the object based onobserved and predicted behavior. In various examples, the trajectoryand/or intent of an object may be determined utilizing techniquesdescribed in U.S. Pat. No. 10,414,395 and/or U.S. patent applicationSer. Nos. 16/151,607, 16/504,147 and/or 15/807,521, incorporated byreference above.

In at least one example, the vehicle computing device(s) 404 may includeone or more system controllers 426, which may be configured to controlsteering, propulsion, braking, safety, emitters, communication, andother systems of the vehicle 402. The system controller(s) 426 maycommunicate with and/or control corresponding systems of the drivesystem(s) 414 and/or other components of the vehicle 402.

As illustrated in FIG. 4, the vehicle computing device(s) 404 mayinclude a warning signal component 428. The warning signal component 428may include the signal emission component 430. In various examples, thesignal emission component 430 may be configured to determine when toemit a warning signal, such as warning signals 202 and 304. In someexamples, the signal emission component 430 may determine to emit awarning signal based on a location associated with the vehicle 402. Insuch examples, the signal emission component 430 may receive locationdata from the localization component 420 and may determine, based on thelocation data, to emit the warning signal. In various examples, thelocation may include an area associated with a school zone, an urbanarea, a business district, a construction zone, and/or other area inwhich pedestrians, scooters, bicyclists, and the like commonly travel.

In various examples, the signal emission component 430 may determine toemit a warning signal based on a speed associated with the vehicle 402.In some examples, the signal emission component 430 may receive anindication of vehicle speed and/or an indication that the vehicle speedis above or below a threshold speed (e.g., 15 miles per hour, 30kilometers per hour, etc.), such as from the perception component 422,and may cause the warning signal to be emitted. In various examples, thewarning signal may include an electric vehicle warning sound, such asthat required by law or regulation to alert objects in the environmentof the electric (quiet) vehicle. In such examples, the signal emissioncomponent 430 may receive an indication that a speed of the vehicle 402is at or below the threshold speed and may cause the warning signal tobe emitted based on the indication.

In various examples, the signal emission component 430 may determine toemit a warning signal based on a detection of an object in anenvironment and/or a determination of relevance of that object to thevehicle. In various examples, the object may be determined to berelevant to the vehicle based on a distance between the object and thevehicle being less than a threshold distance. In some examples, theobject may be relevant based on a determination that a predicted objecttrajectory of the object intersects with a vehicle trajectory associatedwith the vehicle 402. In such examples, the object may be relevant basedon a determination that a conflict (e.g., collision) may exist betweenthe vehicle 402 and the object.

The signal emission component 430 may be configured to determine a setof characteristics associated with the warning signal. In variousexamples, the set of characteristics may be pre-determined (e.g.,pre-determined frequency, volume, luminosity, color, motion, shape,etc.), such as based on the location, speed of the vehicle 402, or thelike. In such examples, the signal emission component 430 may emit thepre-determined warning signal. For example, the warning signal mayinclude an electric vehicle warning sound emitted based on adetermination that the speed of the vehicle 402 is less than 23 milesper hour. The signal emission component 430 may cause the warning signalwith the pre-determined frequency and volume designated for the electricvehicle warning sound to be emitted via one or more emitters 408.

In various examples, the signal emission component 430 may dynamicallydetermine the set of characteristics associated with the warning signal,such as based on real-time conditions. The real-time conditions mayinclude one or more environmental factors (e.g., noise level in theenvironment 100, amount of traffic, proximity to the object 106, etc.),weather conditions (e.g., rain, snow, hail, wind, etc.), vehicularconsiderations (e.g., speed, passengers in the vehicle 104, etc.), dataassociated with the object 106 (e.g., object attribute (e.g.,classification, position (e.g., facing/moving toward the vehicle,facing/moving away from the vehicle, etc.), distance from the vehicle,trajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.), and the like. In such examples, the signal emissioncomponent 430 may receive data associated with the environment, such asfrom the localization component 420 and/or the perception component 422,and may dynamically determine the set of characteristics associated withthe warning signal.

In various examples, the signal emission component 430 may be configuredto determine that one or more of the environmental factors includes anunknown environmental factor. The unknown environmental factor mayinclude a condition in the environment that the signal emissioncomponent 430 is not trained to understand, such as a number ofpedestrians in proximity to the vehicle exceeds a threshold number, anoise level in the environment is above a threshold noise level, or thelike. For example, the signal emission component 430 may determine thatthe vehicle is surrounded by a large crowd of pedestrians. In variousexamples, the signal emission component 430 may determine the set ofcharacteristics based on the unknown environmental factor. In suchexamples, the set of characteristics may include an event-specific setof characteristics based on the unknown environmental factor. In variousexamples, the signal emission component 430 may cause data associatedwith the unknown environmental factor, set of characteristics associatedwith the warning signal, and object reactions to the warning signal tobe stored in the reaction database. In such examples, the data may beutilized to train the system to optimize for an expected (desired)reaction for the unknown environmental factor.

In various examples, the signal emission component 430 may be configuredto determine a direction in which the warning signal is to be emitted.In some examples, the signal emission component 430 may cause thewarning signal to be emitted around the vehicle, such as in alldirections around the vehicle. In some examples, the signal emissioncomponent 430 may cause the warning signal to be emitted in a directionassociated with objects in an environment. For example, the warningsignal may be emitted via speakers on a right side of the vehicle,directed to pedestrians on a sidewalk adjacent a roadway and/or cyclistsin a bicycle lane. In various examples, the signal emission component430 may cause the warning signal to be emitted toward a particularobject, such as a relevant object. In such example, the warning signalmay cause the warning signal to be emitted via an emitter 408 directedtoward the particular object. In some examples, an audio warning signalmay be directed toward the particular object utilizing beam steeringand/or beamformed array techniques.

In various examples, the reaction determination component 432 of thewarning signal component 428 may be configured to determine an objectreaction to the warning signal. The object reaction may include a changein an object trajectory (e.g., speed increase, speed decrease, directionof travel away from the vehicle, etc.), a movement of the head and/orshoulders of the object, a gesture (e.g., a wave, etc.), a footplacement of the object, a positional adjustment to an item the objectholds (e.g., adjusting a position of an electronic device, book,magazine, or other item), and/or any other movement indicative of anobject reacting the first warning signal. In various examples, thereaction determination component 432 may receive sensor data from theperception component 422 and may determine the object reaction based onthe sensor data. In other examples, the reaction determination component432 may receive an indication of the object reaction, such as from theperception component 422. In such examples, the perception component 422may process the sensor data to determine the object reaction.

In various examples, the reaction determination component 432 maycompare the detected object reaction to an expected reaction. In variousexamples, the reaction determination component 432 may access a reactiondatabase 436 (and/or reaction database 456 on computing device(s) 442)to determine the expected reaction. In such examples, the expectedreaction may be stored based on the data associated with the objectand/or the set of characteristics associated with the warning signal.

In various examples, the reaction determination component 432 mayreceive the expected reaction from a machine learning component 434 ormachine learning component 454 of the computing device(s) 442. In suchexamples, the machine learning component 434 and/or 454 may beconfigured to receive data associated with the object and/or the set ofcharacteristics associated with the warning signal and output anexpected reaction. The machine learning components 434 and/or 454 mayinclude one or more models trained utilizing training data comprising aplurality of object reactions to a plurality of warning signals.

In various examples, the machine learning components 434 and/or 454 maybe trained to determine an optimal signal for alerting an object of thepresence of the vehicle. The optimal signal may be based on one or morereal-time considerations present in the environment, such asenvironmental factors, weather conditions, object activity, and the like(as described above). The optimal signal may include a signal that hasthe greatest probability of being successful in alerting a particularobject to the presence and/or operation of the vehicle.

In some examples, the machine learning components 434 and/or 454 may betrained utilizing training data including previously emitted warningsignals, object reactions thereto, and/or associated real-timeconsiderations associated therewith. In such examples, the machinelearning components 434 and/or 454 may be configured to receive inputcomprising real-time considerations and may output an optimal warningsignal (e.g., characteristics associated with an optimal warning signal)and/or an expected reaction thereto. In various examples, the trainingdata may include the previously emitted signals and associated reactionsand/or real-time considerations that were successful in causing objectsto move away from and/or out of the way of the vehicle 402. In suchexamples, the optimal signal output by the machine learning components434 and/or 454 to alert a particular object may include a signal thatresulted in another object with similar attributes to the particularobject reacting according to an expected reaction (e.g., staying out ofthe vehicle path, moving out of the vehicle path, acknowledging thepresence of the vehicle 402, etc.).

Based on the comparison between the object reaction and the expectedreaction, the reaction determination component 432 may be configured todetermine whether the object reaction substantially matches the expectedreaction. In some examples, the object reaction may substantially matchthe expected reaction based on a determination that the object reactionand the expected reaction share a threshold number of actions (e.g.,features). The threshold number of actions may be one or more actions.In some examples, the threshold number of actions may be dynamicallydetermined based on the scenario (e.g., urgency, classification ofobject, vehicle speed, etc.). For example, an urgent warning signal toan intersecting relevant object may include a threshold number of threematching actions to determine that the object reaction substantiallymatches the expected reaction whereas a non-urgent warning signaldirected toward stationary objects located on a sidewalk may include onematching action to determine a substantial match.

In some examples, the object reaction may substantially match theexpected reaction based on a determination that a threshold percentageof actions match between the object reaction and the expected reaction.Continuing the example from above, an urgent warning signal to anintersecting relevant object may include a 90%, match whereas anon-urgent warning signal may include a 50% match.

Responsive to a determination that the object reaction substantiallymatches the expected reaction, the reaction determination component 432may determine that the object has been alerted to the vehicle 402presence and/or operation. In various examples, based on thedetermination of a substantial match, the reaction determinationcomponent 432 may cause data associated with the warning signal and theobject reaction to be stored in the reaction database 436 and/or thereaction database 456. In some examples, based on a determination of asubstantial match, the reaction determination component 432 may providedata associated with the object reaction and the warning signal to themachine learning components 434 and/or 454, such as to train the machinelearning components 434 and/or 454 to output relevant expectedreactions.

Responsive to a determination that the object reaction does notsubstantially match the expected reaction, the reaction determinationcomponent 432 may modify the set of characteristics associated with thewarning signal. In various examples, the reaction determinationcomponent 432 may cause a second (modified) warning signal to beemitted. The second (modified) warning signal may include a second setof characteristics. The second (modified) warning signal may include asignal of a same or a different modality as the first warning signal. Insome examples, the reaction determination component 432 may modify oneor more of a frequency, volume, luminosity, color, motion, and/or shapeof the warning signal to generate the second (modified) warning signal.The reaction determination component 432 may cause the second (modified)warning signal to be emitted via one or more emitters 408, in an updatedattempt to alert the object of the presence and/or operation of thevehicle 402.

In various examples, the reaction determination component 432 maycontinue to modify the sets of characteristics associated with warningsignals until the object reaction substantially matches the expectedreaction. In some examples, the reaction determination component 432 maymodify the sets of characteristics based on a determination of relevanceof the object to the vehicle 402. In such examples, the reactiondetermination component 432 may be configured to determine objectrelevance, such as utilizing the techniques described above. In variousexamples, the reaction determination component 432 may determine whetheran object is relevant prior to generating a modified signal and/orcausing the modified signal to be emitted.

In various examples, the reaction determination component 432 maydetermine that a relevant object is a blocking object. As describedabove with regard to FIG. 3, the blocking object may be in a locationthat at least partially blocks a path of the vehicle 402. In someexamples, responsive to determining that the relevant object is ablocking object, the reaction determination component 432 may send anindication of the blocking object to the object routing determinationcomponent 438. In various examples, the object routing determinationcomponent 438 may be configured to determine whether an area on a roadis clear into which the blocking object may move out of the way of thevehicle 402.

As discussed above, the area may include a location that is not in thevehicle path, the lane associated with the vehicle, and/or the adjacentlane. In some examples, the area may include a location into which theblocking object may move to no longer block progress of the vehicle 402and/or other vehicles/objects traveling in a same direction (in a samelane or on a same roadway) as the vehicle 402. In some examples, thearea may include a location that the operator of the blocking object maybe unable to view, such as based on a viewing path being blocked byanother object.

In various examples, the object routing determination component 438 maysend an indication of the clear area into which the blocking object maymove out of the way of the vehicle to the signal emission component 430.In some examples, the signal emission component 430 may cause a routingsignal to be emitted via the emitter(s) 408. The routing signal mayinclude an indication of the clear area into which the blocking objectmay move, a route thereto, and/or additional information. In variousexamples, the routing signal may include characteristics (e.g.,frequency, volume, luminosity, color, motion, shape, etc.) to indicateto the operator that a path to the area is clear. For example, therouting signal may include a green arrow projected on a surface of theroadway, such as in the lane associated with the clear area. For anotherexample, the routing signal may be projected on the surface of the roadto appear as a set of sequenced flashing lights leading to the cleararea, such as an approach lighting system.

In some examples, the reaction determination component 432 may beconfigured to determine whether the operator reaction matches anexpected reaction. In such examples, the expected reaction may includethe operator of the blocking object following the routing signal (e.g.,moving toward the area, moving out of the path of the vehicle). In someexamples, based on a determination that the operator reactionsubstantially matches the expected reaction and/or the object isirrelevant to the vehicle 402, the reaction determination component 432may send an indication to the signal emission component 430 and/oremitter(s) 408 to stop emitted the routing signal.

As can be understood, the components discussed herein (e.g., thelocalization component 420, the perception component 422, the planningcomponent 424, the one or more system controllers 426, the warningsignal component 428 including the signal emission component 430, thereaction determination component 432, the machine learning component434, the reaction database 436, and the object routing determinationcomponent 438 are described as divided for illustrative purposes.However, the operations performed by the various components may becombined or performed in any other component.

In some instances, aspects of some or all of the components discussedherein may include any models, techniques, and/or machine learningtechniques. For example, in some instances, the components in the memory418 (and the memory 440, discussed below) may be implemented as a neuralnetwork. As described herein, an exemplary neural network is abiologically inspired technique which passes input data through a seriesof connected layers to produce an output. Each layer in a neural networkmay also comprise another neural network, or may comprise any number oflayers (whether convolutional or not). As can be understood in thecontext of this disclosure, a neural network may utilize machinelearning, which may refer to a broad class of such techniques in whichan output is generated based on learned parameters.

In some examples, the vehicle computing device(s) 404 may utilizemachine learning techniques to determine one or more characteristics(e.g., frequency, volume, luminosity, color, shape, motion, etc.) ofwarning signals to be emitted from the vehicle 402. In some examples,one or more data models may be trained to determine the characteristicsof the warning signals based on one or more conditions in theenvironment. The condition(s) may include environmental factors (e.g.,noise level in the environment, amount of traffic, proximity to theobject, etc.), weather conditions (e.g., rain, snow, hail, wind, etc.),data associated with the object (e.g., object attribute (e.g.,classification, position (e.g., facing/moving toward the vehicle,facing/moving away from the vehicle, etc.), distance from the vehicle,trajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.), and the like. In various examples, the data model(s)may be trained to output the characteristics of the warning signal basedat least in part on the condition(s) present in the environment.

Although discussed in the context of neural networks, any type ofmachine learning may be used consistent with this disclosure. Forexample, machine learning techniques may include, but are not limitedto, regression techniques (e.g., ordinary least squares regression(OLSR), linear regression, logistic regression, stepwise regression,multivariate adaptive regression splines (MARS), locally estimatedscatterplot smoothing (LOESS)), instance-based techniques (e.g., ridgeregression, least absolute shrinkage and selection operator (LASSO),elastic net, least-angle regression (LARS)), decisions tree techniques(e.g., classification and regression tree (CART), iterative dichotomiser3 (ID3), Chi-squared automatic interaction detection (CHAID), decisionstump, conditional decision trees), Bayesian techniques (e.g., naïveBayes, Gaussian naïve Bayes, multinomial naïve Bayes, averageone-dependence estimators (AODE), Bayesian belief network (BNN),Bayesian networks), clustering techniques (e.g., k-means, k-medians,expectation maximization (EM), hierarchical clustering), associationrule learning techniques (e.g., perceptron, back-propagation, hopfieldnetwork, Radial Basis Function Network (RBFN)), deep learning techniques(e.g., Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN),Convolutional Neural Network (CNN), Stacked Auto-Encoders),Dimensionality Reduction Techniques (e.g., Principal Component Analysis(PCA), Principal Component Regression (PCR), Partial Least SquaresRegression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS),Projection Pursuit, Linear Discriminant Analysis (LDA), MixtureDiscriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA),Flexible Discriminant Analysis (FDA)), Ensemble Techniques (e.g.,Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, StackedGeneralization (blending), Gradient Boosting Machines (GBM), GradientBoosted Regression Trees (GBRT), Random Forest), SVM (support vectormachine), supervised learning, unsupervised learning, semi-supervisedlearning, etc. Additional examples of architectures include neuralnetworks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, and thelike.

In at least one example, the sensor system(s) 406 may include lidarsensors, radar sensors, ultrasonic transducers, sonar sensors, locationsensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertialmeasurement units (IMUs), accelerometers, magnetometers, gyroscopes,etc.), cameras (e.g., RGB, IR, intensity, depth, time of flight, etc.),microphones, wheel encoders, environment sensors (e.g., temperaturesensors, humidity sensors, light sensors, pressure sensors, etc.), etc.The sensor system(s) 406 may include multiple instances of each of theseor other types of sensors. For instance, the lidar sensors may includeindividual lidar sensors located at the corners, front, back, sides,and/or top of the vehicle 402. As another example, the camera sensorsmay include multiple cameras disposed at various locations about theexterior and/or interior of the vehicle 402. The sensor system(s) 406may provide input to the vehicle computing device(s) 404. Additionallyor alternatively, the sensor system(s) 406 may send sensor data, via theone or more networks 444, to the one or more computing device(s) 442 ata particular frequency, after a lapse of a predetermined period of time,in near real-time, etc.

The vehicle 402 may also include one or more emitters 408 for emittinglight and/or sound, as described above. The emitters 408 in this exampleinclude interior audio and visual emitters to communicate withpassengers of the vehicle 402. By way of example and not limitation,interior emitters may include speakers, lights, signs, display screens,touch screens, haptic emitters (e.g., vibration and/or force feedback),mechanical actuators (e.g., seatbelt tensioners, seat positioners,headrest positioners, etc.), and the like. The emitters 408 in thisexample also include exterior emitters. By way of example and notlimitation, the exterior emitters in this example include lights emittedas a warning signal and/or to signal a direction of travel for an objectand/or the vehicle 402 and/or other indicator of vehicle action (e.g.,indicator lights, signs, light arrays, etc.), and one or more audioemitters (e.g., speakers, speaker arrays, horns, etc.) to audiblycommunicate with pedestrians or other nearby vehicles, one or more ofwhich comprising acoustic beam steering technology.

The vehicle 402 may also include communication connection(s) 410 thatenable communication between the vehicle 402 and one or more other localor remote computing device(s) 442. For instance, the communicationconnection(s) 410 may facilitate communication with other localcomputing device(s) on the vehicle 402 and/or the drive system(s) 414.Also, the communication connection(s) 410 may allow the vehicle tocommunicate with other nearby computing device(s) (e.g., computingdevice(s) 442, other nearby vehicles, etc.) and/or one or more remotesensor system(s) 446 for receiving sensor data.

The communications connection(s) 410 may include physical and/or logicalinterfaces for connecting the vehicle computing device(s) 404 to anothercomputing device or a network, such as network(s) 444. For example, thecommunications connection(s) 410 can enable Wi-Fi-based communicationsuch as via frequencies defined by the IEEE 802.11 standards, shortrange wireless frequencies such as Bluetooth, cellular communication(e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.) or any suitable wired or wirelesscommunications protocol that enables the respective computing device tointerface with the other computing device(s).

In at least one example, the vehicle 402 may include one or more drivesystems 414. In some examples, the vehicle 402 may have a single drivesystem 414. In at least one example, if the vehicle 402 has multipledrive system 414, individual drive systems 414 may be positioned onopposite ends of the vehicle 402 (e.g., the front and the rear, etc.).In at least one example, the drive system(s) 414 may include one or moresensor systems to detect conditions of the drive system(s) 414 and/orthe surroundings of the vehicle 402. By way of example and notlimitation, the sensor system(s) may include one or more wheel encoders(e.g., rotary encoders) to sense rotation of the wheels of the drivesystems, inertial sensors (e.g., inertial measurement units,accelerometers, gyroscopes, magnetometers, etc.) to measure orientationand acceleration of the drive system, cameras or other image sensors,ultrasonic sensors to acoustically detect objects in the surroundings ofthe drive system, lidar sensors, radar sensors, etc. Some sensors, suchas the wheel encoders may be unique to the drive system(s) 414. In somecases, the sensor system(s) on the drive system(s) 414 may overlap orsupplement corresponding systems of the vehicle 402 (e.g., sensorsystem(s) 406).

The drive system(s) 414 may include many of the vehicle systems,including a high voltage battery, a motor to propel the vehicle, aninverter to convert direct current from the battery into alternatingcurrent for use by other vehicle systems, a steering system including asteering motor and steering rack (which can be electric), a brakingsystem including hydraulic or electric actuators, a suspension systemincluding hydraulic and/or pneumatic components, a stability controlsystem for distributing brake forces to mitigate loss of traction andmaintain control, an HVAC system, lighting (e.g., lighting such ashead/tail lights to illuminate an exterior surrounding of the vehicle),and one or more other systems (e.g., cooling system, safety systems,onboard charging system, other electrical components such as a DC/DCconverter, a high voltage j unction, a high voltage cable, chargingsystem, charge port, etc.). Additionally, the drive system(s) 414 mayinclude a drive system controller which may receive and preprocess datafrom the sensor system(s) 406 and to control operation of the variousvehicle systems. In some examples, the drive system controller mayinclude one or more processors and memory communicatively coupled withthe one or more processors. The memory 418 may store one or more modulesto perform various functionalities of the drive system(s) 414.Furthermore, the drive system(s) 414 may also include one or morecommunication connection(s) that enable communication by the respectivedrive system with one or more other local or remote computing device(s)442.

In at least one example, the direct connection 412 may provide aphysical interface to couple the one or more drive system(s) 414 withthe body of the vehicle 402. For example, the direct connection 412 mayallow the transfer of energy, fluids, air, data, etc. between the drivesystem(s) 414 and the vehicle. In some instances, the direct connection412 may further releasably secure the drive system(s) 414 to the body ofthe vehicle 402.

In at least one example, the localization component 420, the perceptioncomponent 422, the planning component 424, the one or more systemcontrollers 426, and the warning signal component 428 and variouscomponents thereof, may process sensor data, as described above, and maysend their respective outputs, over the one or more network(s) 444, tothe computing device(s) 442. In at least one example, the localizationcomponent 420, the perception component 422, the planning component 424,the one or more system controllers 426, and the warning signal component428 may send their respective outputs to the computing device(s) 442 ata particular frequency, after a lapse of a predetermined period of time,in near real-time, etc.

In some examples, the vehicle 402 may send sensor data to the computingdevice(s) 442 via the network(s) 444. In some examples, the vehicle 402may receive sensor data from the computing device(s) 442 and/or one ormore remote sensor systems 446 via the network(s) 444. The sensor datamay include raw sensor data and/or processed sensor data and/orrepresentations of sensor data. In some examples, the sensor data (rawor processed) may be sent and/or received as one or more log files.

The computing device(s) 442 may include processor(s) 448 and a memory440 storing a map component 450, a sensor data processing component 452,a machine learning component 454, and a reaction database 456 (asdescribed above). In some examples, the map component 450 may includefunctionality to generate maps of various resolutions. In such examples,the map component 450 may send one or more maps to the vehicle computingdevice(s) 404 for navigational purposes. In various examples, the sensordata processing component 452 may be configured to receive data from oneor more remote sensors, such as sensor systems 406 and/or remote sensorsystem(s) 446. In some examples, the sensor data processing component452 may be configured to process the data and send processed sensor datato the vehicle computing device(s) 404, such as for use by the warningsignal component 428. In some examples, the sensor data processingcomponent 452 may be configured to send raw sensor data to the vehiclecomputing device(s) 404.

The processor(s) 416 of the vehicle 402 and the processor(s) 448 of thecomputing device(s) 442 may be any suitable processor capable ofexecuting instructions to process data and perform operations asdescribed herein. By way of example and not limitation, the processor(s)416 and 448 may comprise one or more Central Processing Units (CPUs),Graphics Processing Units (GPUs), or any other device or portion of adevice that processes electronic data to transform that electronic datainto other electronic data that may be stored in registers and/ormemory. In some examples, integrated circuits (e.g., ASICs, etc.), gatearrays (e.g., FPGAs, etc.), and other hardware devices may also beconsidered processors in so far as they are configured to implementencoded instructions.

Memory 418 and 440 are examples of non-transitory computer-readablemedia. The memory 418 and 440 may store an operating system and one ormore software applications, instructions, programs, and/or data toimplement the methods described herein and the functions attributed tothe various systems. In various implementations, the memory may beimplemented using any suitable memory technology, such as static randomaccess memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory capable ofstoring information. The architectures, systems, and individual elementsdescribed herein may include many other logical, programmatic, andphysical components, of which those shown in the accompanying figuresare merely examples that are related to the discussion herein.

In some instances, the memory 418 and 440 may include at least a workingmemory and a storage memory. For example, the working memory may be ahigh-speed memory of limited capacity (e.g., cache memory) that is usedfor storing data to be operated on by the processor(s) 416 and 448. Insome instances, the memory 418 and 440 may include a storage memory thatmay be a lower-speed memory of relatively large capacity that is usedfor long-term storage of data. In some cases, the processor(s) 416 and448 cannot operate directly on data that is stored in the storagememory, and data may need to be loaded into a working memory forperforming operations based on the data, as discussed herein.

It should be noted that while FIG. 4 is illustrated as a distributedsystem, in alternative examples, components of the vehicle 402 may beassociated with the computing device(s) 442 and/or components of thecomputing device(s) 442 may be associated with the vehicle 402. That is,the vehicle 402 may perform one or more of the functions associated withthe computing device(s) 442, and vice versa.

FIGS. 5-7 illustrate example processes in accordance with embodiments ofthe disclosure. These processes are illustrated as logical flow graphs,each operation of which represents a sequence of operations that may beimplemented in hardware, software, or a combination thereof. In thecontext of software, the operations represent computer-executableinstructions stored on one or more computer-readable storage media that,when executed by one or more processors, perform the recited operations.Generally, computer-executable instructions include routines, programs,objects, components, data structures, and the like that performparticular functions or implement particular abstract data types. Theorder in which the operations are described is not intended to beconstrued as a limitation, and any number of the described operationsmay be combined in any order and/or in parallel to implement theprocesses.

FIG. 5 depicts an example process 500 for depicts an example process foremitting different signals to warn an object of a potential conflictbetween a vehicle and the object. For example, some or all of theprocess 500 may be performed by one or more components in FIG. 4, asdescribed herein. For example, some or all of the process 500 may beperformed by the vehicle computing device(s) 404.

At operation 502, the process may include detecting, based at least inpart on sensor data, an object in an environment of a vehicle. Thesensor data may include data received from one or more sensors of thevehicle and/or one or more remote sensors, such as sensors mounted inthe environment or mounted on other vehicles. In various examples, avehicle computing system of the vehicle may be configured to determine aclassification (e.g., type) associated with the object.

At operation 504, the process may include emitting a first warningsignal based at least in part on detecting the object. In variousexamples, the first warning signal may be emitted based on adetermination of relevance of the object. In such examples, the vehiclecomputing system may be configured to determine whether an object isrelevant to the vehicle. In various examples, an object relevance may bedetermined utilizing the techniques described in U.S. patent applicationSer. Nos. 16/389,720, 16/417,260, and 16/530,515,” incorporated hereinby reference above.

In some examples, object relevance may be determined based on a distancebetween the object and a drivable surface on which the vehicle operates(e.g., a roadway, lane in which the vehicle operates, etc.). In suchexamples, the object may be determined to be relevant based on thedistance being equal to or less than a threshold distance (e.g., 23inches, 5 feet, 5 meters, etc.). In various examples, the thresholddistance may be determined based on the classification associated withthe object and/or an object activity. For example, a first thresholddistance associated with a walking pedestrian may be 2 meter and asecond threshold distance associated with a running pedestrian may be 4meters.

In various examples, the object may be determined to be relevant basedon an object trajectory associated therewith. In such examples, thevehicle computing system may be configured to determine a predictedobject trajectory (e.g., object trajectory), such as based on the sensordata. As discussed above, the object trajectory may be determinedaccording to the techniques described in U.S. patent application Ser.Nos. 16/151,607, 16/504,147, and 15/807,521, incorporated herein byreference above.

In various examples, the object may be determined to be relevant to thevehicle based on an intersection between the object trajectory and avehicle trajectory. In some examples, the object may be relevant basedon predicted locations of the object and the vehicle on the respectivetrajectories. In some examples, the object may be relevant to thevehicle based on a determination that a predicted future object locationassociated with the object traveling on the object trajectory is withina threshold distance (e.g., 3 feet, 9 feet, 1.5 meters, 3.3 meters,etc.) of a predicted future vehicle location associated with the vehicletraveling on the vehicle trajectory.

In various examples, the first warning signal may include an audiosignal and/or a visual signal. The first warning signal may include afirst set of characteristics, such as frequency, volume, luminosity,color, shape, motion, or the like. In some examples, the first set ofcharacteristics may include a pre-determined set of characteristics. Insuch examples, the first warning signal may include a baseline warningsignal associated with alerting objects of the presence and/or operationof the vehicle. In various examples, the first set of characteristicsmay be determined dynamically, such as based on one or more real-timeconditions associated with the environment. The real-time conditions mayinclude data associated with the object (e.g., object attribute (e.g.,classification, position (e.g., facing/moving toward the vehicle,facing/moving away from the vehicle, etc.), distance from the vehicle,trajectory, etc.), object activity (e.g., walking, running, riding ascooter, (e.g., a particular activity implied by an object trajectory,such as based on speed, etc.), reading a book, talking on a phone,viewing data on an electronic device, interacting with another vehicle,interacting with another object (e.g., talking to another person,looking into a stroller, etc.), eating, drinking, operating a sensoryimpairment device (e.g., cane, hearing aid, etc.), listening toheadphones, etc.), environmental factors (e.g., noise level in theenvironment, amount of traffic, road conditions, etc.), weatherconditions (e.g., rain, snow, hail, wind, etc.), vehicularconsiderations (e.g., speed, passengers in the vehicle, etc.), and thelike.

In various examples, the first warning signal may be emitted in adirection associated with the object. For example, the vehicle computingsystem may cause the first warning signal to be emitted via emitterssubstantially facing the object. In some examples, the first warningsignal may be directed at the object, such as in a beamformed array.

At operation 506, the process may include determining whether the objectreacts according to an expected reaction to the (first) warning signaland that the object remains relevant to the vehicle. In variousexamples, the vehicle computing system may verify the relevance of theobject to the vehicle prior to or concurrently with determining whetherthe object reacts according to the expected reaction.

In various examples, the vehicle computing system may be configured todetermine an object reaction to the first warning signal, based on thesensor data. In some examples, the reaction may include a change in theobject trajectory (e.g., speed increase, speed decrease, direction oftravel away from the vehicle, etc.), a movement of the head and/orshoulders of the object, a gesture (e.g., a wave, etc.), a footplacement of the object, a positional adjustment to an item the objectholds (e.g., adjusting a position of an electronic device, book,magazine, or other item), and/or any other movement indicative of anobject reacting the first warning signal.

In various examples, the vehicle computing system may compare the objectreaction to an expected reaction associated with the first warningsignal. In various examples, the vehicle computing system may beconfigured to determine the expected reaction based on one or morecharacteristics of the first warning signal (e.g., volume, frequency,luminosity, color, shape, motion, etc.) and/or data associated with theobject (e.g., object attribute (e.g., classification, position (e.g.,facing/moving toward the vehicle, facing/moving away from the vehicle,etc.), distance from the vehicle, trajectory, etc.), object activity(e.g., walking, running, riding a scooter, (e.g., a particular activityimplied by an object trajectory, such as based on speed, etc.), readinga book, talking on a phone, viewing data on an electronic device,interacting with another vehicle, interacting with another object (e.g.,talking to another person, looking into a stroller, etc.), eating,drinking, operating a sensory impairment device (e.g., cane, hearingaid, etc.), listening to headphones, etc.). In some examples, thevehicle computing system may access a database of expected reactions todetermine the expected reaction associated with the first warningsignal. In such examples, the expected reactions in the database may bestored based at least in part on the data associated with the objectand/or characteristic(s) of the first warning signal. In variousexamples, the vehicle computing system may determine an expectedreaction utilizing machine learning techniques. In such examples, amodel may be trained utilizing training data including a plurality ofwarning signals and detected reactions thereto.

Based on the comparison between the object reaction and the expectedreaction, the vehicle computing system may determine whether the objectreacts according to the expected reaction (e.g., whether a substantialmatch exists between the object reaction and the expected reaction).

Based on a determination that the object reacts according to theexpected reaction (e.g., “Yes” at 506), the process, at operation 508may include storing the object reaction in a reaction database. In someexamples, the reaction database may be used for future object reactioncomparisons, such as to increase a confidence in a reaction to the firstwarning signal, to train the machine learned model, or the like. Invarious examples, the data associated with the first warning signal, theobject reaction and/or the real-time considerations associated with theenvironment may be used to train a machine learned model for selectingan optimal signal for notifying (e.g., alerting) an object.

Based on a determination that the object does not react according to theexpected reaction (e.g., “No” at 506), the process, at operation 510,may include emitting a second warning signal based at least in part onthe object reaction. In some examples, the second signal may include amodification to the first warning signal. In such examples, the secondsignal may include a signal with a different frequency, volume,luminosity, color, shape, motion, and the like, as compared to the firstwarning signal.

In some examples, the vehicle computing system may determine a secondset of characteristics based on a pre-determined modification to thefrequency, volume, luminosity, color, shape, motion, and the like. Forexample, subsequent warning signals may include increasing volumes, suchthat a first warning signal may include an audio signal emitted at 50decibels, the second warning signal at 60 decibels, and so on. Invarious examples, the vehicle computing system may determine a secondset of characteristics based on one or more real-time conditions. Asdiscussed above, the real-time conditions may include environmentalfactors, weather conditions, vehicular considerations, data associatedwith the object, and the like.

In various examples, the second warning signal may be emitted in adirection associated with the object. For example, the vehicle computingsystem may cause the second warning signal to be emitted via emitterssubstantially facing the object. In some examples, the second warningsignal may be directed at the object, such as in a beamformed array.

In various examples, the vehicle computing system may store dataassociated with the first warning signal, the object reaction, and/orreal-time considerations based on a determination that the object doesnot react according to an expected reaction (“No” at operation 510). Insome examples, the data may be utilized to compare the relativeeffectiveness of different warning signals, such as to determine anoptimized signal for a given scenario.

After emitting the second warning signal, the process may include againdetermining whether the object reacts according to an expected reactionto the warning signal and that the object remains relevant to thevehicle, such as that illustrated at operation 506. In various examples,the vehicle computing system may continuously modify (e.g., iterativelymodify) the warning signal until the vehicle computing system determinesthat the object reacts according to the expected reaction or determinesthat the object is irrelevant to the vehicle. In some examples, thevehicle computing system may modify the warning signal a pre-determinednumber of times and/or until the set of characteristics associated witha warning signal includes a maximum volume, frequency, and/orluminosity. In some examples, the vehicle computing system may cause thelast modified warning signal to be emitted until the object is no longerrelevant to the vehicle. In some examples, the vehicle computing systemmay cause the last modified warning signal to be emitted for apre-determined amount of time (e.g., 30 seconds, 2 minutes, etc.).

FIG. 6 depicts an example process 600 for emitting warning signals basedat least in part on a location of a vehicle and detection of an objectthat is relevant to the vehicle. For example, some or all of the process600 may be performed by one or more components in FIG. 4, as describedherein. For example, some or all of the process 600 may be performed bythe vehicle computing device(s) 404.

At operation 602, the process may include determining a speed and/orlocation of the vehicle in an environment. In various examples, avehicle computing system may determine the speed and/or the location ofthe vehicle based on data provided by one or more sensors of thevehicle.

At operation 604, the process may include determining whether the speedand/or location is associated with warning signal emission. In variousexamples, a speed of the vehicle may be associated with the warningsignal emission. In such examples, based on the speed of the vehiclebeing below a threshold speed, the vehicle computing system may causethe vehicle to emit a warning signal.

In various examples, a location of the vehicle may be associated withthe warning signal emission. In some examples, the location may beassociated with a classification of object (e.g., pedestrians,bicyclists, etc.). In various examples, the location may be associatedwith a school zone, proximity to a playground, downtown area, businessdistrict, construction zone, popular cycling route, or the like.

In various examples, the location may be associated with theclassification of objects, zones, etc. based on a time of day, day ofthe week, date (e.g., holiday, season, etc.). In such examples, thevehicle computing system may determine a time of day, day of the week,date, etc. and determine whether the location is associated with warningsignal emission. For example, the vehicle computing system may operatein a school zone associated with pedestrians. Based on a determinationthat the day and/or date is associated with a school day, the vehiclecomputing system may determine that the location is associated withwarning signal emission.

Based on a determination that the location is not associated with speedand/or location associated with warning signal emission (“No” atoperation 604), at operation 606, the process may include determiningwhether a relevant object is detected in the environment.

As discussed above, the object may be detected based on sensor datareceived from one or more sensors of the vehicle and/or one or moreremote sensors. In various examples, the vehicle computing system maydetermine whether the object is relevant to the vehicle. As discussedabove, a determination of relevance may be based on a distance betweenthe object and the vehicle, a distance between the object and a vehiclepath (e.g., drivable surface, lane, etc. associated with the vehiclepath), one or more object trajectories, a vehicle trajectory, and thelike.

Based on a determination that a relevant object is not detected in thearea (“No” at operation 606), the process may include determining thespeed and/or location of the vehicle in the environment, such as thatdescribed with regard to operation 602.

Based on a determination that the speed and/or location of the vehicleis associated with warning signal emission (“Yes” at operation 604) orthat a relevant object is detected in the environment (“Yes” atoperation 606), the process may include, at operation 608, emitting afirst signal (e.g., first warning signal) based in part on the speed,location, and/or relevant object. The first signal may include an audioand/or a visual warning signal. The first signal may include a first setof characteristics (e.g., frequency, volume, luminosity, color, shape,motion, etc.). The first set of characteristics may include one or morepre-determined characteristics and/or one or more dynamically determinedcharacteristics. The pre-determined characteristic(s) may be based onthe speed, location, and/or relevant object (e.g., classification,proximity, etc.). The dynamically determined characteristic(s) may bebased on one or more real-time conditions in the environment (e.g., dataassociated with the object, environmental factors, weather conditions,vehicular considerations, etc.).

In various examples, the first signal may be emitted in a directionassociated with the object. For example, the vehicle computing systemmay cause the first signal to be emitted via emitters substantiallyfacing the object. In some examples, the first signal may be directed atthe object, such as in a beamformed array.

At operation 610, the process may include determining whether an objectreacts (to the first signal) according to an expected reaction. Invarious examples, the vehicle computing system may determine an objectreaction, such as based on sensor data. The object reaction may includea change (or lack thereof) in an object trajectory (e.g., speedincrease, speed decrease, direction of travel away from the vehicle,etc.), a movement of the head and/or shoulders of the object, a gesture(e.g., a wave, etc.), a foot placement of the object, a positionaladjustment to an item the object holds (e.g., adjusting a position of anelectronic device, book, magazine, or other item), and/or any othermovement indicative of an object reacting the first signal.

The vehicle computing system may compare the object reaction to theexpected reaction to determine whether the object reacts according tothe expected reaction. In some examples, the computing system may accessa database of expected reactions to determine the expected reaction. Invarious examples, the expected reaction may be stored in the databasebased on data associated with the object, characteristic(s) of the firstsignal, or the like. In some examples, the vehicle computing system maydetermine the expected reaction utilizing machine learning techniques.In such examples, the vehicle computing system may input the dataassociated with the object and/or characteristic(s) of the first signalinto a machine learned model trained to determine expected reactions ofobject and may receive an output of an expected reaction.

As discussed above, the object may react according to the expectedreaction based on a substantial match between the (observed, detected)object reaction and the expected reaction. The vehicle computing systemmay determine a substantial match based on a number of actions (e.g.,features) and/or a percentage of actions between the object reaction andthe expected reaction matching.

Based on a determination that the object reacts according to an expectedreaction (“Yes” at operation 610), the process, at operation 612, mayinclude storing the object reaction in a reaction database, such asdatabase 122. In some examples, the database may be used for futureobject reaction comparisons, such as to increase a confidence in anobject reaction to the first signal, to train the machine learned model,or the like.

Based on a determination that the object does not react according to theexpected reaction (“No” at operation 610), the process, at operation614, may include determining whether the object remains relevant to thevehicle. The determination of continued relevance may be based onrelevance determination techniques described above, such as in thedescription of operation 606.

In various examples, the vehicle computing system may store dataassociated with the first signal, the object reaction, and/or real-timeconsiderations based on a determination that the object does not reactaccording to an expected reaction (“No” at operation 610). In someexamples, the data may be utilized to compare the relative effectivenessof different warning signals, such as to determine an optimized signalfor a given scenario.

Based on a determination that the object is irrelevant to the vehicle(“No” at operation 614), the process may include, at operation 602,determining the speed and/or location of the vehicle in the environment.

Based on a determination that the object is relevant to the vehicle(“Yes” at operation 614), the process may include, at operation 616,emitting a second signal based at least in part on the object reaction.The second signal may include an audio and/or visual signal emitted toalert the object of the vehicle presence and/or operation. The secondsignal may include a same or a different modality(ies) as the firstsignal. The second signal may include a second set of characteristics.In various examples, the second set of characteristics may include oneor more characteristics that are different from the first set ofcharacteristics. In some examples, the vehicle computing system maymodify the first set of characteristics to generate the second signal(e.g., second set of characteristics).

In various examples, the second signal may be emitted in a directionassociated with the object. For example, the vehicle computing systemmay cause the second signal to be emitted via emitters substantiallyfacing the object. In some examples, the second signal may be directedat the object, such as in a beamformed array.

Based at least in part on emitting the second signal, the process mayinclude, at operation 610, determining whether the object reacts (to thesecond signal) according to an expected reaction. In various examples,the vehicle computing system may continue to modify the emitted signalsuntil the object becomes irrelevant to the vehicle or the object reactsaccording to the expected reaction. In some examples, the vehiclecomputing system may modify the signal a pre-determined number of times(e.g., 7 times, 10 times, etc.). In such examples, the vehicle computingsystem may cease modifying the emitted signals. In some examples, thevehicle computing system may modify the signal for a pre-determinedperiod of time. In such examples, the vehicle computing system may ceasemodifying the signals after the period of time has expired.

FIG. 7 depicts an example process 700 for emitting at least one of awarning signal or a routing signal based on a determination that anobject is blocking a path of a vehicle. For example, some or all of theprocess 700 may be performed by one or more components in FIG. 4, asdescribed herein. For example, some or all of the process 700 may beperformed by the vehicle computing device(s) 404.

At operation 702, the process may include determining that an object inan environment is blocking the vehicle path. A vehicle computing systemmay determine that the object is a blocking object based on sensor datareceived from one or more sensors of a vehicle and/or remote sensor(s)in the environment. In various examples, the vehicle path may include adrivable area of a roadway associated with a route from a currentlocation of the vehicle to a destination. In some examples, the drivablearea may include the width of the vehicle and/or a buffer distance oneither side of the vehicle (e.g., 12 centimeters, 6 inches, 1 foot,etc.).

In various examples, the vehicle computing system may determine theobject is blocking the vehicle path based on a determination that anobject location associated with the object is at least partially withinthe vehicle path. In various examples, the vehicle computing system maydetermine that the object is blocking the vehicle path based on adetermination that the vehicle is not able to proceed around the objectin a lane associated with the vehicle path.

At operation 704, the process may include emitting a first signal basedon the object blocking the vehicle path. The first signal may include anaudio and/or a visual warning signal. The first signal may include afirst set of characteristics (e.g., frequency, volume, luminosity,color, shape, motion, etc.). The first set of characteristics mayinclude one or more pre-determined characteristics and/or one or moredynamically determined characteristics. The pre-determinedcharacteristic(s) may be based on the speed, location, and/or relevantobject (e.g., classification, proximity, etc.). The dynamicallydetermined characteristic(s) may be based on one or more real-timeconditions in the environment (e.g., data associated with the object,environmental factors, weather conditions, vehicular considerations,etc.).

In various examples, the first signal may be emitted in a directionassociated with the object. For example, the vehicle computing systemmay cause the first signal to be emitted via emitters substantiallyfacing the object. In some examples, the first signal may be directed atthe object, such as in a beamformed array.

At operation 706, the process may include determining whether the objectreacts (to the first signal) according to an expected reaction. Invarious examples, the vehicle computing system may determine an objectreaction, such as based on sensor data. The object reaction may includea change (or lack thereof) in an object trajectory (e.g., speedincrease, speed decrease, direction of travel away from the vehicle,etc.), a movement of the head and/or shoulders of the object, a gesture(e.g., a wave, etc.), a foot placement of the object, a positionaladjustment to an item the object holds (e.g., adjusting a position of anelectronic device, book, magazine, or other item), and/or any othermovement indicative of an object reacting the first signal.

The vehicle computing system may compare the object reaction to theexpected reaction to determine whether the object reacts according tothe expected reaction. In some examples, the computing system may accessa database of expected reactions to determine the expected reaction. Invarious examples, the expected reaction may be stored in the databasebased on data associated with the object, characteristic(s) of the firstsignal, or the like. In some examples, the vehicle computing system maydetermine the expected reaction utilizing machine learning techniques.In such examples, the vehicle computing system may input the dataassociated with the object and/or characteristic(s) of the first signalinto a machine learned model trained to determine expected reactions ofobject and may receive an output of an expected reaction.

As discussed above, the object may react according to the expectedreaction based on a substantial match between the (observed, detected)object reaction and the expected reaction. The vehicle computing systemmay determine a substantial match based on a number of actions (e.g.,features) and/or a percentage of actions between the object reaction andthe expected reaction matching.

Based on a determination that the object does not react according to theexpected reaction (“No” at operation 706), the process, at operation708, may include emitting a second signal toward the object. The secondsignal may include an audio and/or a visual warning signal. The secondsignal may include a second set of characteristics (e.g., frequency,volume, luminosity, color, shape, motion, etc.). The second set ofcharacteristics may include one or more pre-determined characteristicsand/or one or more dynamically determined characteristics. Thepre-determined characteristic(s) may be based on the speed, location,and/or relevant object (e.g., classification, proximity, etc.). Thedynamically determined characteristic(s) may be based on one or morereal-time conditions in the environment (e.g., data associated with theobject, environmental factors, weather conditions, vehicularconsiderations, etc.).

In various examples, the second signal may be emitted in the directionassociated with the object. For example, the vehicle computing systemmay cause the second signal to be emitted via emitters facing theobject. In some examples, the second signal may be directed at theobject, such as in a beamformed array.

Based on a determination that the object reacts according to an expectedreaction (“Yes” at operation 706), the process, at operation 710, mayinclude determining whether an area for the object to move out of thevehicle path is identified. In some examples, the area may include alocation that is not in the vehicle path, a lane associated with thevehicle, and/or an adjacent lane. In such examples, the area may includea location to which the blocking object may move to no longer blockprogress of the vehicle and/or other vehicles/objects traveling in thelane and/or the adjacent lane. In some examples, the area may include asize large enough for the object to move and no longer block progress ofthe vehicle and/or other vehicles/objects. In some examples, the areamay include a location that the operator of the object may be unable toview, such as based on a viewing path being blocked by another object.

Based on a determination that the area for the object to move out of thevehicle path exists (“Yes” at operation 710), the process, at operation712, may include emitting a third signal including an indication of thearea. In some examples, the third signal may include an indication of anobject path (route) out of the vehicle path. In some examples, the thirdsignal may indicate to the operator of the object that the area existsand is clear. In various examples, the third signal may include a symbolor other indicator, such as an arrow, to indicate to the operator of theobject a location associated with the area. In various examples, thesymbol or other indicator may be projected on a drivable surfaceproximate the object and/or the area. In some examples, the symbol orother indicator may include a holographic image projected in view of theoperator of the object.

At operation 714, the process may include determining that the object isirrelevant to the vehicle. Additionally, the vehicle computing systemmay determine that the object is irrelevant based on a determinationthat the area for the object of move out of the path does not exist(“No” at operation 710). In various examples, a determination that theobject is irrelevant to the vehicle may be based on a determination thatthe object is no longer blocking the vehicle path. In such examples, thevehicle computing system may determine that the object has moved into ortoward the area (e.g., according to the third signal) and/or anotherarea out of the vehicle path.

At operation 716, the process may include controlling the vehicleaccording to the vehicle path. In various examples, the vehicle controlaccording to the vehicle path may be based on traffic rules, laws, etc.For example, the vehicle computing system may determine that, by thetime the object is no longer blocking the vehicle path, that a trafficlight has turned red. Based on the determination that the traffic lightis red, the vehicle may maintain a position and wait for the trafficlight to turn green.

EXAMPLE CLAUSES

A: A vehicle comprising: a sensor; an emitter; one or more processors;and one or more computer-readable media storing instructions that, whenexecuted, configure the vehicle to: determine, based at least in part onsensor data from the sensor, an object in an environment associated withthe vehicle; determine, based at least in part on the sensor data, anobject trajectory associated with the object; determine, based at leastin part on the object trajectory, that the object is relevant to aprogress of the vehicle; emit, via the emitter, a first signal based atleast in part on determining that the object is relevant to the progressof the vehicle, the first signal comprising a first characteristic;determine, based at least in part on the sensor data, an object reactionto the first signal; and based at least in part on the object reaction,emit a second signal, wherein the second signal comprises a secondcharacteristic different from the first characteristic.

B: A vehicle as paragraph A describes, wherein emitting the secondsignal is further based at least in part on determining that the objectreaction differs from an expected reaction, wherein the object reactionis a first object reaction and the expected reaction is a first expectedreaction, and wherein the instructions further cause the vehicle to:determine a second object reaction to the second signal; and based atleast in part on the second object reaction, store data associated withthe second object reaction in a database.

C: A vehicle as either one of paragraphs A or B describe, wherein: thefirst characteristic comprises at least one of: one or more firstfrequencies; one or more first volumes; one or more first luminosities;one or more first colors; one or more first shapes; or one or more firstmotions; and the second characteristic comprises at least one of: one ormore second frequencies; one or more second volumes; one or more secondluminosities; one or more second colors one or more second shapes; orone or more second motions.

D: A vehicle as any one of paragraphs A-C describe, wherein at least oneof the first characteristic or the second characteristic is based atleast in part on an activity associated with the object, the activitycomprising at least one of: listening to headphones; viewing data on amobile device; reading a book; talking on a mobile phone; eating;drinking; a particular activity implied by a predicted trajectory;operating a sensory impairment device; a head of the object facing adirection away from a location associated with the vehicle; interactingwith another vehicle in the environment; or interacting with anotherobject in proximity to the object.

E: A vehicle as any one of paragraphs A-C describe, wherein theinstructions further cause the vehicle to determine the expectedreaction based at least in part on at least one of: machine learningtechniques; or expected reaction data stored in a database, wherein theexpected reaction is associated with at least one of: the firstcharacteristic; a classification of the object; a position of theobject; or an object activity.

F: A computer-implemented method comprising: detecting, based on sensordata from a sensor on a vehicle, an object in an environment, the objectcomprising an object attribute; causing, based at least in part on theobject attribute, a first signal to be emitted via an emitter of avehicle at a first time, the first signal comprising a firstcharacteristic; determining, based at least in part on additional sensordata from the sensor, an object reaction of the object at a second timeafter the first time; and based on the object reaction, causing a secondsignal to be emitted via the emitter of the vehicle, the second signalcomprising a second characteristic.

G: A computer-implemented method as paragraph F describes, wherein thesensor data is first sensor data, the method further comprising:determining that the object is relevant to a progress of the vehicle;and causing at least one of the first signal or the second signal to beemitted in a direction associated with the object based at least in parton determining that the object is relevant to the progress of thevehicle.

H: A computer-implemented method as either one of paragraphs F or Gdescribe, wherein causing the second signal to be emitted is furtherbased on determining that the object continues to impede progress of thevehicle after the second time.

I: A computer-implemented method as any one of paragraphs F-H describe,wherein at least one of the first characteristic or the secondcharacteristic is based at least in part on at least one of: anenvironmental factor in the environment; a weather condition in theenvironment a location of the vehicle in the environment; a speed of thevehicle in the environment; an activity associated with the object; aposition of the object relative to the vehicle; a time of day in whichthe vehicle is operating; a time of year in which the vehicle isoperating; or a day in a week in which the vehicle is operating.

J: A computer-implemented method as paragraph I describes, wherein theactivity comprises one or more of: listening to headphones; viewing dataon a mobile device; reading a book; talking on a mobile phone; eating;drinking; a particular activity implied by a predicted trajectory;operating a sensory impairment device; a head of the object facing adirection away from a location associated with the vehicle; interactingwith another vehicle in the environment; or interacting with anotherobject in proximity to the object.

K: A computer-implemented method as any one of paragraphs F-I describe,wherein causing the second signal to be emitted is further based atleast in part on determining that the object reaction differs from anexpected reaction and wherein the object reaction is a first objectreaction and the expected reaction is a first expected reaction, themethod further comprising: determining a second object reaction of theobject to the second signal; and storing data associated with at leastone of the second signal or the second object reaction in a databasebased at least in part on the second object reaction.

L: A computer-implemented method as any one of paragraphs F-K describe,wherein the emitter comprises at least one of: a speaker; a light; or aprojector.

M: A computer-implemented method as any one of paragraphs F-L describe,further comprising iteratively emitting additional signals until atleast one of: determining that the object is not associated with thevehicle; determining that a timer associated with a warning signal hasexpired; or determining that a number of warning signals emitted meetsor exceeds a threshold number.

N: A computer-implemented method as any one of paragraphs F-M describe,further comprising: determining that the object is at least partiallyblocking a vehicle path associated with the vehicle; identifying alocation for the object to move, the location being outside the vehiclepath and clear of other objects; and based at least in part onidentifying the location, causing a third signal to be emitted via asecond emitter, wherein the third signal provides an indication to theobject of the location for the object to move.

O: A system or device comprising: a processor; and a non-transitorycomputer-readable medium storing instructions that, when executed, causea processor to perform a computer-implemented method as any one ofparagraphs F-M describe.

P: A system or device comprising: a means for processing; and a meansfor storing coupled to the means for processing, the means for storingincluding instructions to configure one or more devices to perform acomputer-implemented method as any one of paragraphs F-M describe.

Q: One or more non-transitory computer-readable media storinginstructions that, when executed, cause a vehicle to perform operationscomprising: detecting, based at least in part on sensor data from asensor, an object in an environment, the object comprising an objectattribute; based at least in part on the object attribute, emitting afirst signal via an emitter of a vehicle at a first time, the firstsignal comprising a first characteristic; determining, based at least inpart on additional sensor data from the sensor, an object reaction of anobject at a second time after the first time; and based on the objectreaction emitting a second signal via the emitter of the vehicle, thesecond signal comprising a second characteristic.

R: One or more non-transitory computer-readable media as paragraph Qdescribes, the operations further comprising: determining that theobject is relevant to the progress of the vehicle; and causing at leastone of the first signal or the second signal to be emitted in adirection associated with the object based at least in part ondetermining that the object is relevant to the progress of the vehicle.

S: One or more non-transitory computer-readable media as either one ofparagraphs Q or R describe, the operations further comprising:determining an activity associated with the object; and determining thefirst characteristic based at least in part on the activity, wherein theactivity comprises one or more of: listening to headphones; viewing dataon a mobile device; reading a book; talking on a mobile phone; eating;drinking; a particular activity implied by a predicted trajectory;operating a sensory impairment device; a head of the object facing adirection away from a location associated with the vehicle; interactingwith another vehicle in the environment; or interacting with anotherobject in proximity to the object.

T: One or more non-transitory computer-readable media as any one ofparagraphs Q-S describe, the operations further comprising determiningthe second characteristic utilizing a machine learned model, the machinelearned model trained based at least in part on previously emittedsignals which caused additional objects having similar attributes toperform an action to unblock the vehicle.

U: One or more non-transitory computer-readable media as any one ofparagraphs Q-T describe, wherein: the object attribute comprises anobject trajectory, the object reaction comprises a modification to atleast one of a speed or a direction associated with the objecttrajectory, and emitting the second signal is based at least in part ondetermining that the modification to the at least one of the speed orthe direction is less than a threshold modification associated with anexpected reaction.

V: One or more non-transitory computer-readable media as paragraph Udescribes, the operations further comprising further comprisingiteratively emitting additional signals until at least one of:determining that the object is not relevant to a progress of thevehicle; determining that a timer associated with a warning signal hasexpired; or determining that a number of warning signals emitted meetsor exceeds a threshold number.

While the example clauses A-V described above are described with respectto one particular implementation, it should be understood that, in thecontext of this document, the content of the example clauses A-V mayalso be implemented via a method, device, system, a computer-readablemedium, and/or another implementation.

CONCLUSION

While one or more examples of the techniques described herein have beendescribed, various alterations, additions, permutations and equivalentsthereof are included within the scope of the techniques describedherein.

In the description of examples, reference is made to the accompanyingdrawings that form a part hereof, which show by way of illustrationspecific examples of the claimed subject matter. It is to be understoodthat other examples can be used and that changes or alterations, such asstructural changes, can be made. Such examples, changes or alterationsare not necessarily departures from the scope with respect to theintended claimed subject matter. While the steps herein may be presentedin a certain order, in some cases the ordering may be changed so thatcertain inputs are provided at different times or in a different orderwithout changing the function of the systems and methods described. Thedisclosed procedures could also be executed in different orders.Additionally, various computations that are herein need not be performedin the order disclosed, and other examples using alternative orderingsof the computations could be readily implemented. In addition to beingreordered, the computations could also be decomposed intosub-computations with the same results.

What is claimed is:
 1. A vehicle comprising: a sensor; an emitter; one or more processors; and one or more computer-readable media storing instructions that, when executed, configure the vehicle to: determine, based at least in part on sensor data from the sensor, an object in an environment associated with the vehicle; determine, based at least in part on the sensor data, an object trajectory associated with the object; determine, based at least in part on the object trajectory, that the object is relevant to a progress of the vehicle; emit, via the emitter, a first signal based at least in part on determining that the object is relevant to the progress of the vehicle, the first signal comprising a first characteristic; determine, based at least in part on the sensor data, an object reaction to the first signal; based at least in part on the object reaction differing from an expected reaction, emit a second signal, wherein the second signal comprises a second characteristic different from the first characteristic; and store data associated with the first signal and the object reaction in a database.
 2. The vehicle of claim 1, wherein the instructions further cause the vehicle to: determine a second object reaction to the second signal; and based at least in part on the second object reaction, store data associated with the second object reaction in the database.
 3. The vehicle of claim 1, wherein: the first characteristic comprises at least one of: one or more first frequencies; one or more first volumes; one or more first luminosities; one or more first colors; one or more first shapes; or one or more first motions; and the second characteristic comprises at least one of: one or more second frequencies; one or more second volumes; one or more second luminosities; one or more second colors; one or more second shapes; or one or more second motions.
 4. The vehicle of claim 1, wherein at least one of the first characteristic or the second characteristic is based at least in part on an activity associated with the object, the activity comprising at least one of: listening to headphones; viewing data on a mobile device; reading a book; talking on a mobile phone; eating; drinking; a particular activity implied by a predicted trajectory; operating a sensory impairment device; a head of the object facing a direction away from a location associated with the vehicle; interacting with another vehicle in the environment; or interacting with another object in proximity to the object.
 5. The vehicle of claim 1, wherein the instructions further cause the vehicle to determine the expected reaction based at least in part on at least one of: machine learning techniques; or expected reaction data stored in a database, wherein the expected reaction is associated with at least one of: the first characteristic; a classification of the object; a position of the object; or an object activity.
 6. A method comprising: detecting, based on sensor data from a sensor on a vehicle, an object in an environment, the object comprising an object attribute; causing, based at least in part on the object attribute, a first signal to be emitted via an emitter of a vehicle at a first time, the first signal comprising a first characteristic; determining, based at least in part on additional sensor data from the sensor and at a second time after the first time an object reaction of the object to the first signal; based on the object reaction differing from an expected reaction, causing a second signal to be emitted via the emitter of the vehicle, the second signal comprising a second characteristic; and storing data associated with the first signal and the object reaction in a database.
 7. The method of claim 6, wherein the sensor data is first sensor data, the method further comprising: determining that the object is relevant to a progress of the vehicle; and causing at least one of the first signal or the second signal to be emitted in a direction associated with the object based at least in part on determining that the object is relevant to the progress of the vehicle.
 8. The method of claim 6, wherein causing the second signal to be emitted is further based on determining that the object continues to impede progress of the vehicle after the second time.
 9. The method of claim 6, wherein at least one of the first characteristic or the second characteristic is based at least in part on at least one of: an environmental factor in the environment; a weather condition in the environment; a location of the vehicle in the environment; a speed of the vehicle in the environment; an activity associated with the object; a position of the object relative to the vehicle; a time of day in which the vehicle is operating; a time of year in which the vehicle is operating; or a day in a week in which the vehicle is operating.
 10. The method of claim 9, wherein the activity comprises one or more of: listening to headphones; viewing data on a mobile device; reading a book; talking on a mobile phone; eating; drinking; a particular activity implied by a predicted trajectory; operating a sensory impairment device; a head of the object facing a direction away from a location associated with the vehicle; interacting with another vehicle in the environment; or interacting with another object in proximity to the object.
 11. The method of claim 6, further comprising: determining a second object reaction of the object to the second signal; and storing data associated with at least one of the second signal or the second object reaction in the database based at least in part on the second object reaction.
 12. The method of claim 6, wherein the emitter comprises at least one of: a speaker; a light; or a projector.
 13. The method of claim 6, further comprising iteratively emitting additional signals until at least one of: determining that the object is not associated with the vehicle; determining that a timer associated with a warning signal has expired; or determining that a number of warning signals emitted meets or exceeds a threshold number.
 14. The method of claim 6, further comprising: determining that the object is at least partially blocking a vehicle path associated with the vehicle; identifying a location for the object to move, the location being outside the vehicle path and clear of other objects; and based at least in part on identifying the location, causing a third signal to be emitted via a second emitter, wherein the third signal provides an indication to the object of the location for the object to move.
 15. One or more non-transitory computer-readable media storing instructions that, when executed, cause a vehicle to perform operations comprising: detecting, based at least in part on sensor data from a sensor, an object in an environment, the object comprising an object attribute; based at least in part on the object attribute, emitting a first signal via an emitter of a vehicle at a first time, the first signal comprising a first characteristic; determining, based at least in part on additional sensor data from the sensor and at a second time after the first time, an object reaction of the object to the first signal; based at least in part on the object reaction differing from an expected reaction, emitting a second signal via the emitter of the vehicle, the second signal comprising a second characteristic; and storing data associated with the first signal and the object reaction in a database.
 16. The one or more non-transitory computer-readable media of claim 15, the operations further comprising: determining that the object is relevant to the progress of the vehicle; and causing at least one of the first signal or the second signal to be emitted in a direction associated with the object based at least in part on determining that the object is relevant to the progress of the vehicle.
 17. The one or more non-transitory computer-readable media of claim 15, the operations further comprising: determining an activity associated with the object; and determining the first characteristic based at least in part on the activity, wherein the activity comprises one or more of: listening to headphones; viewing data on a mobile device; reading a book; talking on a mobile phone; eating; drinking; a particular activity implied by a predicted trajectory; operating a sensory impairment device; a head of the object facing a direction away from a location associated with the vehicle; interacting with another vehicle in the environment; or interacting with another object in proximity to the object.
 18. The one or more non-transitory computer-readable media of claim 15, the operations further comprising determining the second characteristic utilizing a machine learned model, the machine learned model trained based at least in part on previously emitted signals which caused additional objects having similar attributes to perform an action to unblock the vehicle.
 19. The one or more non-transitory computer-readable media of claim 15, wherein: the object attribute comprises the object trajectory, the modification comprises a change to at least one of a speed or a direction associated with the object trajectory, and emitting the second signal is based at least in part on determining that the change to the at least one of the speed or the direction is less than a threshold change associated with an expected reaction.
 20. The one or more non-transitory computer-readable media of claim 19, the operations further comprising further comprising iteratively emitting additional signals until at least one of: determining that the object is not relevant to a progress of the vehicle; determining that a timer associated with a warning signal has expired; or determining that a number of warning signals emitted meets or exceeds a threshold number. 